ICV: Real-Time Computer Vision for Drilling Rigs

In short: in 2020–2021 I led the dvhb R&D team that designed and built ICV, a real-time computer vision system for an oilfield services company. A server on the rig itself watched the drilling floor through CCTV cameras, read the rig's sensors and warned the crew while an operation was going wrong, before a human error turned into an accident at thousands of metres of depth.

The problem#

Drilling crews follow strict procedures, and small slips in them are expensive. The client picked two to start with:

• Drill string without movement. When the crew leaves the drill string standing in the slips for too long, it can stick in the well. Freeing stuck pipe can stop drilling for days.
• Casing make-up. Every casing connection has to be screwed with the right number of turns. A bad connection fails thousands of metres down, where it is very costly to fix.

Both are problems of time: an alert helps only while the crew can still act. So the system had to see and decide in real time, on the rig.

Before building anything, we studied the client's incident database and interviewed drilling supervisors, drillers and assistant drillers about how these incidents happen.

What we built#

We called each rule the system checks a marker: a business rule over what the cameras and sensors see.

Marker 1: drill string without movement#

An SSD object detector, trained on 4 classes of 700 frames each, recognises the state of the equipment around the rotary table: slips open or closed, slips closed and covered, and the spider. A region of interest and scene segmentation keep the model on the rotary table and away from the rest of the floor. A log analyzer turns frame-by-frame detections into time intervals, removes noise and raises an alert when the string stands still for longer than the set limit, such as 3 or 5 minutes.

Marker 2: casing turns#

To check a connection, the system has to count how many times the pipe turns while the crew screws it. But casing pipes carry no marks that show a full turn. I suggested using the dirt instead: every pipe has its own pattern of dirt on its surface, so the system learns that pattern and counts a turn each time it comes round again. Tracking, image stabilisation, feature detection and optical flow follow the pipe through the turns.

Architecture#

Each rig gets its own well-site server, so detection runs in real time next to the cameras and keeps working when the internet doesn't:

• Inputs: PoE cameras record to a video recorder, which streams over RTSP. Rig sensors connect over Modbus.
• Well-site server: a video grabber feeds the CV module (object detection plus post-processing that removes impossible combinations of detections). A sensor grabber publishes sensor data over MQTT with Protobuf. The log analyzer combines both, and a well manager applies the rig's settings and sends alerts.
• On the rig floor: sound and light alarms for the crew, and a well-site dashboard for the supervisor with every event and its video clip.
• Cloud server: collects events from all rigs over MQTT into PostgreSQL, and runs the managers' dashboard, mobile apps and remote notifications by SMS, email, push and webhooks.

New markers had to be possible without new code, so we also designed a visual marker composer, where a rule is a graph of detections, sensor values and conditions.

Behind the models sits a data pipeline, and it was a challenge of its own: the rig cameras gave us terabytes of raw CCTV footage to process and annotate. Raw video goes to labelling in CVAT, then to GPU training and testing, all automated. More GPU capacity cut a training run from 18 hours to 2. A video archive with search, filters, automatic tagging, clip cutting and thumbnails let the team find and label new situations fast.

The team#

I led a team of eight at dvhb: a product designer, a data scientist and ML engineer with a background in oil and gas, a principal backend architect for high-load computer vision and ML systems, backend and frontend engineers, a DevOps engineer and a data manager.

Lessons from a working rig#

A drilling rig is a hard place for real-time computer vision:

• Image quality. Steam, snow, mud on the lens and night shifts, all of which the training data had to include.
• Connectivity. The rig had mobile internet with a monthly data cap and low bandwidth. Video stays on the well-site server; only events and short clips go to the cloud.
• Hardware. Overheating, GPU driver failures and power cuts. Remote monitoring and debugging tools were as important as the models.

Working on something similar?#

If you need computer vision that works in real time, at the edge, in harsh conditions or with poor connectivity, tell me about your case and we'll see whether I can help.

👉 Book an intro call

All computer vision projects