# How to add AI camera to BrainFrame **URL:** \ **Category:** Questions\ **Created:** [February 24, 2020, 1:30am UTC](https://forum.aotu.ai/t/how-to-add-ai-camera-to-brainframe/25 "2020-02-24T01:30:44Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![george](https://avatars.discourse-cdn.com/v4/letter/g/71e660/32.png) [@george](https://forum.aotu.ai/u/george)\ **Post date:** [February 24, 2020, 1:30am UTC](https://forum.aotu.ai/t/how-to-add-ai-camera-to-brainframe/25/1 "2020-02-24T01:30:44Z")
I am trying to add SimCam1s AI camera to VisionCapsules runtime environment. What I have done so far includes: 1. Add the metadata containing the results of the deep learning algorithm to the SEI part of the IDR frame, and then pack it into RTP packets. 2.Use GStreamer pipeline to verify the correctness of metadata and video data Rtsp source --\> h264 file pipeline to verify metadata: gst-launch-1.0 -e rtspsrc location=rtsp://SIMCAM:183HX9@192.168.1.107/live ! rtph264depay ! ‘video/x-h264, stream-format=(string)byte-stream’ ! filesink location=file.h264 Rtsp source --\> decode --\> mp4 file pipeline to verify video data: gst-launch-1.0 -e rtspsrc location=rtsp://SIMCAM:183HX9@192.168.1.107/live! rtph264depay ! ‘video/x-h264, stream-format=(string)byte-stream’ ! decodebin ! avenc\_mpeg4 ! mp4mux ! filesink location=file.mp4 Does the brainframe support adding new appsink, such as sub\_sink? I plan to write a gstreamer plugin to extract sei data from h264 video stream. ---
**Author:** ![alex.thiel](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.aotu.ai/alex.thiel/32/48_2.png) [@alex.thiel](https://forum.aotu.ai/u/alex.thiel)\ **Post date:** [February 24, 2020, 1:41am UTC](https://forum.aotu.ai/t/how-to-add-ai-camera-to-brainframe/25/2 "2020-02-24T01:41:02Z")
Thank you for your questions. Currently BrainFrame doesn’t support extracting AI metadata, but we’re interested in adding this feature. Our plan is to extract SEI data in BrainFrame by default and pass it to Capsules “process\_frame” method along with the numpy frame. Perhaps if your GStreamer plugin for extracting SEI metadata can be shipped with BrainFrame once we see that it works well with the rest of the system? I’ll discuss this with the rest of the team. ---
**Author:** ![george](https://avatars.discourse-cdn.com/v4/letter/g/71e660/32.png) [@george](https://forum.aotu.ai/u/george)\ **Post date:** [March 7, 2020, 1:21pm UTC](https://forum.aotu.ai/t/how-to-add-ai-camera-to-brainframe/25/3 "2020-03-07T13:21:26Z")
Hello Alex,I am a novice to GStreamer, so it took me a week to learn this framework. At present I have written a simply H264SeiFilter to extracting our metadata in SEI. In this plugin, I implemented a tcp service to send metadata to the python tcp client. 1. What should I do next to pass it to Capsules “process\_frame” method? 2. How does Brainframe get data from GStreamer’s appsink？ 3. Could you give me some advice？ ---
**Author:** ![alex.thiel](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.aotu.ai/alex.thiel/32/48_2.png) [@alex.thiel](https://forum.aotu.ai/u/alex.thiel)\ **Post date:** [March 9, 2020, 5:30am UTC](https://forum.aotu.ai/t/how-to-add-ai-camera-to-brainframe/25/4 "2020-03-09T05:30:10Z")
Hi george, thank you for your work thus far. Right now it’s our turn to add metadata extraction to the BrainFrame side. Would you be willing to share your H264SeiFilter Gst plugin and an example of how to run it with your SEI metadata example video? From there, we can add the plugin to BrainFrame deployments and add the feature to the OpenVisionCapsule spec. ---
**Author:** ![george](https://avatars.discourse-cdn.com/v4/letter/g/71e660/32.png) [@george](https://forum.aotu.ai/u/george)\ **Post date:** [March 9, 2020, 8:28am UTC](https://forum.aotu.ai/t/how-to-add-ai-camera-to-brainframe/25/5 "2020-03-09T08:28:08Z")
Of course I am happy to share it, but this plugin is just a simple demo, currently only extracting the metadata of our camera. I am currently installing brainframe, and plan to share a more comprehensive plugin after completing the overall verification ---
**Author:** ![george](https://avatars.discourse-cdn.com/v4/letter/g/71e660/32.png) [@george](https://forum.aotu.ai/u/george)\ **Post date:** [March 12, 2020, 2:13am UTC](https://forum.aotu.ai/t/how-to-add-ai-camera-to-brainframe/25/6 "2020-03-12T02:13:48Z")
Hi Alex,thanks for your help, BrainFrame Server & Client have been installed. How can I register my gstreamer plugin to BrainFrame Client? I try to copy my gstreamer plugin to the path “./v0\_23\_2\_client/lib/gstreamer-1.0”. Then I run BrainFrame Client with custom pipelines: `rtspsrc location={url} ! rtph264depay name="buffer_src" ! video/x-h264, stream-format=byte-stream ! h264seifilter ! decodebin ! videoconvert ! appsink name="main_sink"` ![config](https://us1.discourse-cdn.com/flex019/uploads/aotu/original/1X/21c0575ffe040aac9a9c03b2d5b2f666af29b026.png) ![err](https://us1.discourse-cdn.com/flex019/uploads/aotu/original/1X/98c81a804dbad2928438574a020d734391afb59d.png) By the way, the pipeline without custom plugin works: > rtspsrc location={url} ! rtph264depay name=“buffer\_src” ! video/x-h264, stream-format=byte-stream ! decodebin ! videoconvert ! appsink name=“main\_sink” ---
**Author:** ![alex.thiel](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.aotu.ai/alex.thiel/32/48_2.png) [@alex.thiel](https://forum.aotu.ai/u/alex.thiel)\ **Post date:** [March 12, 2020, 5:24am UTC](https://forum.aotu.ai/t/how-to-add-ai-camera-to-brainframe/25/7 "2020-03-12T05:24:49Z")
Hi George! ### 1: Creating a custom BrainFrame Server with your plugin The plugin needs to be added on the BrainFrame Server container as well. To do that, we can edit the server container by making a custom image on your local machine (for testing purposes). Create a file and call it Dockerfile, with this directory structure ```auto . ├── Dockerfile └── h264seifilter.so ``` Then, inside of Dockerfile: ```auto FROM dilililabs/brainframe:0.23.2 WORKDIR /usr/local/lib/gstreamer-1.0/ ADD h264seifilter.so . # Always end the file in the /brainframe directory so that the # brainframe_server executable is in the PATH WORKDIR /brainframe ``` Then, to build the container, simply run ```auto docker build --tag custom_brainframe:0.23.2 . ``` ### 2. Running the Custom BrainFrame Server Now, we need to add a docker-compose.override.yml so that you can run BrainFrame Server with your own custom image instead of the default `dilililabs/brainframe:0.23.0` image. To do that, go to your docker-compose.yml that you downloaded from [our downloads section](https://aotu.ai/docs/downloads/), and make a directory like this: ```auto . ├── docker-compose.override.yml ├── docker-compose.yml └── license_file ``` Inside of `docker-compose.override.yml` put the following: ```auto version: '2.4' services: api: image: custom_brainframe:0.23.2 ``` Now, to start the BrainFrame server, you just have to run the following in that directory ```auto docker-compose down && docker-compose up ``` ### Helpful Commands If you need to debug inside of the container, you can always enter it by doing the following command: ```auto docker run -it --entrypoint bash dilililabs/brainframe:0.23.2 ``` ---
**Author:** ![george](https://avatars.discourse-cdn.com/v4/letter/g/71e660/32.png) [@george](https://forum.aotu.ai/u/george)\ **Post date:** [March 13, 2020, 4:07am UTC](https://forum.aotu.ai/t/how-to-add-ai-camera-to-brainframe/25/8 "2020-03-13T04:07:49Z")
Hi Alex, great! It works now. ![Screenshot from 2020-03-13 11-54-52](https://us1.discourse-cdn.com/flex019/uploads/aotu/original/1X/2cfcca8feb74d3af74cde1e34d1042a9ed2771ee.jpeg) I have some question: 1. Why should the custom plugin been added in both Sides (Server & Client side) ? 2. Which side The custom pipeline work on,Server or Client? 3. It seems that the default pipeline don’t support H264 NALU with SEI, so I have to filter SEI data from the raw video stream. ---
**Author:** ![tyler.compton](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.aotu.ai/tyler.compton/32/27_2.png) [@tyler.compton](https://forum.aotu.ai/u/tyler.compton)\ **Post date:** [March 13, 2020, 6:02am UTC](https://forum.aotu.ai/t/how-to-add-ai-camera-to-brainframe/25/9 "2020-03-13T06:02:34Z")
Hi George! > [@george](#): > > Why should the custom plugin been added in both Sides (Server & Client side) ? > [@george](#): > > Which side The custom pipeline work on,Server or Client? Custom pipelines are used by both the server and the client, so if you use the h264seifilter plugin on your custom pipeline, it has to be available in both places. > [@george](#): > > It seems that the default pipeline don’t support H264 NALU with SEI, so I have to filter SEI data from the raw video stream. In what way are you finding it unsupported? It’s true that BrainFrame doesn’t currently extract SEI data automatically. Are you finding that SEI data interferes with stream decoding? ---
**Author:** ![george](https://avatars.discourse-cdn.com/v4/letter/g/71e660/32.png) [@george](https://forum.aotu.ai/u/george)\ **Post date:** [March 13, 2020, 6:10am UTC](https://forum.aotu.ai/t/how-to-add-ai-camera-to-brainframe/25/10 "2020-03-13T06:10:41Z")
> [@tyler.compton](#): > > In what way are you finding it unsupported? It’s true that BrainFrame doesn’t currently extract SEI data automatically. Are you finding that SEI data interferes with stream decoding? I can open the rtsp stream with SEI data using vlc player without any error,however brainframe client displays connecting. When I filter the SEI data, It works. ---
**Author:** ![george](https://avatars.discourse-cdn.com/v4/letter/g/71e660/32.png) [@george](https://forum.aotu.ai/u/george)\ **Post date:** [March 13, 2020, 6:39am UTC](https://forum.aotu.ai/t/how-to-add-ai-camera-to-brainframe/25/11 "2020-03-13T06:39:12Z")
I have sent my source code to you from WeChat. Hope it is helpful for your plan to integrate extraction of SEI data by default. If anything I could help , please let me know.