I have a blazor application that I am using to have a conversation with MistralAI.
When prompted, MistralAI will use Server-Sent Events to "stream" data back.
On my .RAZOR page, after a users enters their query, I have this:
<div><EditForm Model="this" OnValidSubmit="@QueryAI"><input type="text" /></EditForm></div>@code { StreamingQueryResponse streamingResponse = await service.getStreamingResponseAsync(conversation); String word = streamingResponse.choices[0].delta.content;}I have a service, registered with the application that calls MistralAI:
public async Task<StreamingQueryResponse> getStreamingResponseAsync(Conversation conversation){ try { string requestString = JsonSerializer.Serialize(conversation); StringContent content = new StringContent(requestString, Encoding.UTF8, "application/json"); using HttpResponseMessage httpResponse = await opService.StreamingPostAsync("http://my.endpoint/", content); httpResponse.EnsureSuccessStatusCode(); using Stream responseStream = await httpResponse.Content.ReadAsStreamAsync(); bool go = true; do { var buffer = new byte[1024]; var bytesRead = await responseStream.ReadAsync(buffer, 0, buffer.Length); if (bytesRead == 0) break; string response = Encoding.UTF8.GetString(buffer, 0, bytesRead); try {As var jsonString = JsonSerializer.Deserialize<StreamingQueryResponse>(response); if (jsonString.choices != null) { // At this point, I am getting all of the tokens (words) from the LLM. // I just need to figure out how to send those to the .RAZOR page without "return"-ing. // Console.WriteLine(jsonString.choices[0].delta.content); go = (jsonString.choices[0].finish_reason == null); return jsonString; } } catch (Exception ex) { Console.WriteLine(ex.Message); } } while (go); } catch (Exception ex) { Console.WriteLine(ex.Message); } return null;}The opService:
public async Task<HttpResponseMessage> StreamingPostAsync(string uri, HttpContent content = null){ HttpRequestMessage request = new HttpRequestMessage { RequestUri = new Uri(uri, UriKind.Absolute), Method = HttpMethod.Post, Content = content }; request.Headers.Add("Accept","application/json"); HttpClient client = _clientFactory.CreateClient(uri); HttpResponseMessage httpResponse = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead); return httpResponse;}As my comment in the code snippet above states, when I put a Console.WriteLine() within the "if" statement of my service, I see the stream coming back from MistralAI. However, I cannot figure out how to get that stream back to the .RAZOR page so I can display the "live" chat.
When I use return I, obviously, just get the first word back, and that's it.
How do I go about doing this?