Querys that return large # of results can freeze / crash the app #137

Activity

1. matthewmturner commented on Sep 12, 2024

   

   matthewmturner

   on Sep 12, 2024

   CollaboratorAuthor

   More actions

   @alamb FYI pagination also relevant for local context
2. alamb commented on Sep 12, 2024

   

   alamb

   on Sep 12, 2024

   Last edited by alamb

   Contributor

   More actions

   @alamb FYI pagination also relevant for local context

   I agree

   One way to do this might be to wrap the SendableRecordBatchStream in an object that buffers past results and fetches next result on demand. Something like this perhaps

   struct SendableRecordBatchStream {
     // currently executing stream
     inner: SendableRecordBatchStream,
     // any batches that have been buffered so far
     batches: Vec<RecordBatch>,
     // current batch being shown
     current_batch: usize,
   }
   
   impl SendableRecordBatchStream {
     /// return the next batch in the stream
     async fn next(&self) -> &RecordBatch {
       self.current_batch += 1;
       self.current()
     }
   
     /// go to current_index
    async fn current(&self) -> &RecordBatch{
       while self.batches.len()+1 < current_batch {
         // fetch next batch from inner and put into self.batches
       }
       self.batches[self.current_batch]
     }
   }

   ❤️React with ❤️1Reacted by Matthew Turner
3. matthewmturner

   linked a pull request that will close this issueAdd pagination to query Execution #141on Sep 13, 2024
4. matthewmturner

   linked a pull request that will close this issueStart setting up flightsql pagination #163on Oct 5, 2024
5. matthewmturner

   linked a pull request that will close this issueFeat/sql pagination 2 #157on Oct 5, 2024
6. matthewmturner

   closed this as completedon Oct 5, 2024