Google recently released Gemini Pro 1.5, a groundbreaking upgrade to their series of AI models, boasting a context size of 1,000,000 tokens—far surpassing previous models like Claude 2.1 and gpt-4-turbo. The most exciting feature is the ability to use video as input, yielding impressive results even with complex footage like a fast-paced, partially obscured bookshelf video. However, safety filters can be triggered by certain content, requiring adjustments for optimal performance. Gemini 1.5 can analyze up to 1 hour of video, breaking it down into frames for processing, with each frame costing around 258 tokens. Overall, this advancement showcases the potential of AI technology in extracting structured data from various media formats.
https://simonwillison.net/2024/Feb/21/gemini-pro-video/