Animal behavior is a major discipline in biology. The majority of studies have focused on those animals that are easiest to see and hear (ie, birds). However, the majority of mammals are nocturnal and elusive (ie, rodents), making it very difficult to examine their behavior. New technologies such as thermal imaging allow us to remotely eavesdrop on the behaviors of nocturnal rodents. However, analyzing thermal video data is difficult because, although hundreds of hours of video data contain a wealth of information, the data cannot be processed efficiently by human observers. Moreover, associated demographic information needs to be applied to the video data to make meaningful observations about patterns of behavior.
In our field studies, we have collected more than 1600 hours of data from a live oak riparian habitat in coastal California where two species of Peromyscus (P. californicus and P. boylii) that differ in body size are the dominant residents. The object of this study was to process these videos to extract meaningful behavioral data.
The track data obtained with computer vision techniques animals is processed to identify tracks of the same inspanidual. Then we use size, speed, and movement patterns to assign individuals to particular species. We validated our output by comparing behavioral patterns with output from those same videos that were completely annotated by a human observer.
The video is first analyzed using background subtraction and blob tracking. Frame-by-frame, each moving blob's position is recorded.
The tracking data is processed and the speed of the blobs is computed using data about the focal area from a spreadsheet.
After the Track-joining step the tracking data and the notes of a human observer are compiled on an HTML page.
Background subtraction and blob tracking were written using the libraries OpenCV
and cvBlob. The processing and output routines were written in Python.
In the background subtraction step the mice (and other moving objects, such as rats) are isolated from the complex background, so that they can be seen as white moving blobs on a black background.
We use the algorithm described in Liyuan Li, Weimin Huang, Irene Y.H. Gu, and Qi Tian
Foreground Object Detection from Videos Containing Complex Background,
ACM MM2003 implemented in OpenCV.
Unfortunately this resulting video quite often still has minor disturbances we remove in the clean up step. In this step we also
dilate the image, such that blobs that belong to the same individual are merged.
In the third step the blobs are recognized and their movements is tracked.
The tracking data for each frame of the video the tracking function returns a list of all blobs together with their position and a bounding box for the blob.
The blobs are identified with
A linear-time component-labeling algorithm using contour tracing technique
by Fu Chang, Chun-Jen Chen and Chi-Jen Lu (Computer Vision and Image Understanding, 2003)
mplemented in the C++ library cvBlob by Cristóbal Carnero Liñán.
When a list of blobs is generated from a video, it is a list of times with what blobs were present (and where) during that time. We reorganized this list into a collection of blobs and used a method termed track-joining to make the list more efficient and weed out errant tracks. The motivation for track-joining is that some of the tracks may actually belong to one blob, but were not correctly tracked by the blob-tracking piece of the program, for instance if an animal went under a branch, or in some other way disappeared
from the view of the camera. To achieve this, we pitted each track against every other track and ranked their joining based on the four characteristics of positional distance between the tracks, temporal distance between the tracks, average speeds of the two tracks and the direction the blob was moving where one track left off and the direction of the other track. The ratings for each were between -5 to 5, and weighted appropriately based on what is more important for the two tracks to be joined.
The videos we worked with for the development stage of this program were short (~1-5 minutes in length). This is not representative of the videos that the program was designed to work with, which are about three hours long. When a small video was processed, the small amount of activity allowed us to easily understand the graphical output. However, when a large video was processed, the increased activity in the video caused the graphical output to be very jumbled and hard to understand. To alleviate this problem, and make the image easier to understand, we broke up the output into several images. The points at which the output is broken up are determined by the amount of
activity on the video at a given time, or by notes from human observation. This is illustrated in Figure 7.
To verify the data obtained from the tracking and track joining steps we output it together with the notes of a human observer
on a HTML page. We visualize the computed imformation as a plot of tracks on top of a still from the video. This is realized as an
HTML5 canvas. For a user selected time interval we show the tracks and the observers comments.
The conversion of the data is done by a Python script.
Displaying ~0 seconds to ~459 seconds. Tracks in blue are outside this range.
Scale: 100px = 2.484m
Comments:
0:0:0-0:4:18 No movement
0:4:19-0:4:19 Bird enters bottom right screen and exits in top right part of screen
0:4:20-0:7:15 No movement or activity
0:7:16-0:7:39 Mouse appears from foliage bottom center screen - stays stationary
Average Speeds:
No. 0: 12.7853155014 p/f
Displaying ~459 seconds to ~699 seconds. Tracks in blue are outside this range.
Scale: 100px = 2.484m
Comments:
0:7:40-0:7:55 Woodrat enters from top right corner, heads to bottom left corner, goes down by stationary mouse, exits at 7:55
0:7:56-0:9:11 Mouse stays stationary
0:9:12-0:9:12 Stationary mouse disappears
0:9:13-0:11:39 No movement or activity
Average Speeds:
No. 1: 0.875849625044 p/f
Displaying ~699 seconds to ~824 seconds. Tracks in blue are outside this range.
Scale: 100px = 2.484m
Comments:
0:11:40-0:12:46 Mouse (mice?) appear in center bottom; 3 appear at different intervals; nest?
0:12:47-0:13:37 Mouse enters from upper right corner; heads down the screen; makes its way to center of screen; checks out grey box in center; goes to left side of screen
0:13:38-0:13:38 Stationary mouse re-appears
0:13:39-0:13:44 Moving mouse makes it way toward stationary mouse
Average Speeds:
No. 2: 0.519310760787 p/f
No. 3: 0.541244259968 p/f
No. 4: 0.346265911902 p/f
No. 5: 0.254764303422 p/f
No. 6: 0.305432605783 p/f
No. 7: 0.202648635322 p/f
No. 8: 0.642119860302 p/f
No. 9: 0.404011034181 p/f
No. 10: 1.02157951956 p/f
Displaying ~824 seconds to ~923 seconds. Tracks in blue are outside this range.
Scale: 100px = 2.484m
Comments:
0:13:45-0:13:45 Stationary mouse fades out; moving mouse still going toward where stationary mouse was
0:13:46-0:13:53 Moving mouse continues to walk around area until it disappears into the foliage near where the other mouse faded
0:13:54-0:13:56 No movement or activity
0:13:57-0:15:23 Another mouse appears in the upper right corner; moves around in that area, becomes stationary for a little bit, goes in and out of the foliage
Average Speeds:
No. 10: 1.02157951956 p/f
No. 11: 0.499785294492 p/f
No. 12: 0.46478385319 p/f
No. 13: 0.337416508398 p/f
No. 14: 0.167428580391 p/f
No. 15: 0.285830382215 p/f
No. 16: 0.324406593437 p/f
No. 17: 0.179223506667 p/f
No. 18: 0.140425596696 p/f
No. 19: 0.219856588889 p/f
No. 20: 0.220002881168 p/f
No. 21: 0.388336556063 p/f
No. 22: 0.368739390601 p/f
Displaying ~923 seconds to ~983 seconds. Tracks in blue are outside this range.
Scale: 100px = 2.484m
Comments:
0:15:24-0:15:32 Mouse reappears in bottom center and moves to the bottom left; upper right mouse still active and moving
0:15:33-0:15:34 Mouse at bottom exits; mouse on top right still active
0:15:35-0:16:13 Mouse in upper right continues to move around and become stationary in intervals in the same area
0:16:14-0:16:23 Mouse disappears under the foliage
Average Speeds:
No. 22: 0.368739390601 p/f
No. 23: 0.172311977768 p/f
No. 24: 0.992891615101 p/f
No. 25: 0.193913748309 p/f
No. 26: 0.189289309755 p/f
No. 27: 0.0835125252073 p/f
No. 28: 0.390758252257 p/f
Displaying ~983 seconds to ~1042 seconds. Tracks in blue are outside this range.
Scale: 100px = 2.484m
Comments:
0:16:24-0:16:55 Mouse reappears, then goes down screen and to the left then back up
0:16:56-0:17:11 Mouse becomes stationary, but still visible
0:17:12-0:17:17 New mouse enters upper right screen near stationary mouse
0:17:18-0:17:22 Old mouse (the one that was stationary) moves to the right and becomes stationary again and the new moouse moves down towards it
Average Speeds:
No. 29: 0.3413398576 p/f
No. 30: 0.260033409624 p/f
No. 31: 0.184681539124 p/f
No. 32: 0.457031966382 p/f
No. 33: 0.620317716465 p/f
Displaying ~1042 seconds to ~1122 seconds. Tracks in blue are outside this range.
Scale: 100px = 2.484m
Comments:
0:17:23-0:17:25 The two mice meet; the old mouse follows/chases the new mouse
0:17:26-0:18:11 The old mouse disappears into the foliage and the new mouse continues slowly down screen to the left and center
0:18:12-0:18:36 The old mouse reappears and walks around in the upper right area of the screen; the new mouse continues to walk around the middle screen
0:18:37-0:18:42 The old mouse disappears into the foliage again and the new mouse continues to walk around the middle screen
Average Speeds:
No. 33: 0.620317716465 p/f
No. 34: 0.155127269484 p/f
No. 35: 0.404063140917 p/f
No. 36: 0.294031802998 p/f
No. 37: 0.323703072593 p/f
No. 38: 0.359383916955 p/f
No. 39: 0.353404803547 p/f
No. 40: 0.296007382199 p/f
No. 41: 0.309122707711 p/f
Displaying ~1122 seconds to ~1143 seconds. Tracks in blue are outside this range.
Scale: 100px = 2.484m
Comments:
0:18:43-0:18:43 The old mouse reappears
0:18:44-0:18:59 Mouse appears in lower left screen; the three mice move about; lower mouse (the one that just entered) disappears
0:19:0-0:19:0 Middle mouse ('new mouse') disappears
0:19:1-0:19:3 Upper mouse ('old mouse') still visible
Average Speeds:
No. 41: 0.309122707711 p/f
No. 42: 0.857330693696 p/f
No. 43: 0.480988305864 p/f
No. 44: 0.895682644928 p/f
Displaying ~1143 seconds to ~1174 seconds. Tracks in blue are outside this range.
Scale: 100px = 2.484m
Comments:
0:19:4-0:19:9 Lower mouse reappears and moves around; upper mouse stationary
0:19:10-0:19:15 Lower mouse disappears; upper mouse still stationary
0:19:16-0:19:22 Lower mouse reappears and moves bottom left; upper mouse still stationary
0:19:23-0:19:34 Lower mouse disappears; upper mouse begins to move down a little; middle mouse still has not been seen
Average Speeds:
No. 45: 0.486567815265 p/f
No. 46: 0.686280485441 p/f
No. 47: 0.502906850705 p/f
No. 48: 0.229627006358 p/f
Displaying ~1174 seconds to ~1222 seconds. Tracks in blue are outside this range.
Scale: 100px = 2.484m
Comments:
0:19:35-0:19:35 Upper mouse disappears
0:19:36-0:19:48 No activity or movement
0:19:49-0:19:58 Upper mouse reappears and moves up screen and down screen in the area
0:19:59-0:20:22 Middle mouse reappears and moves to the right and around the middle screen; upper mouse continues moving around the upper right corner area
Average Speeds:
No. 48: 0.229627006358 p/f
No. 49: 0.190436889845 p/f
No. 50: 0.3256016132 p/f
No. 51: 0.319603520606 p/f
No. 52: 0.374301785276 p/f
No. 53: 0.188156938543 p/f
No. 54: 0.241461339137 p/f
Displaying ~1222 seconds to ~1265 seconds. Tracks in blue are outside this range.
Scale: 100px = 2.484m
Comments:
0:20:23-0:20:27 Lower mouse reappears and remains stationary; other two move
0:20:28-0:20:51 Upper mouse disappears; middle moves a lot in its area; lower moves a little, but sometimes fades into the foliage but reappears quickly and sometimes remains stationary
0:20:52-0:21:3 Upper reappears; all three mice move, but the middle moves the most and the upper and lower mice tend to stay stationary for a time or only move a little
0:21:4-0:21:5 Middle mouse disappears; lowe mouse getting closer to bottom edge of screen; upper mouse still stationary
Average Speeds:
No. 53: 0.188156938543 p/f
No. 54: 0.241461339137 p/f
No. 55: 0.199391466073 p/f
No. 56: 0.442290765177 p/f
No. 57: 0.216487398764 p/f
No. 58: 0.844937928222 p/f
No. 59: 0.460261272474 p/f
No. 60: 0.256767282076 p/f
No. 61: 0.298170191014 p/f
No. 62: 0.381004515093 p/f
No. 63: 0.462758858433 p/f
No. 64: 0.132680413762 p/f
Displaying ~1265 seconds to ~1292 seconds. Tracks in blue are outside this range.
Scale: 100px = 2.484m
Comments:
0:21:5-0:21:8 Upper mouse moves a little in its area and the lower mouse is on edge of screen and fading out
0:21:9-0:21:9 Lower mouse exits; upper mouse moves in its area
0:21:10-0:21:12 Upper mouse moves to the bottom right
0:21:13-0:21:32 Upper mouse disappears (by trap) as the middle mouse reappears; middle mouse moves down left and becomes stationary
Average Speeds:
No. 65: 0.531101880201 p/f
No. 66: 0.389780255349 p/f
Displaying ~1292 seconds to ~1508 seconds. Tracks in blue are outside this range.
Scale: 100px = 2.484m
Comments:
0:21:33-0:23:51 Upper mouse reappears by trap and seems to take wood off of top of trap (?) and seems to be tugging on the trap but then becomes stationary; Middle mouse remains stationary
0:23:52-0:24:2 Middle mouse moves upper left, becomes stationary, then moves down left a litte; Upper mouse still stationary by trap
0:24:3-0:24:14 Middle mouse disappears and Upper mouse remains stationary
0:24:15-0:25:8 Middle mouse reappears and moves to upper screen, sometimes stopping and becoming stationary, then moves right across screen to the upper right corner where the Upper mouse is still stationary by the trap
Average Speeds:
No. 67: 0.141221888973 p/f
No. 68: 0.254740921402 p/f
No. 69: 0.313515087548 p/f
No. 70: 0.533095248768 p/f
No. 71: 0.359469893503 p/f
No. 72: 0.176385029145 p/f
No. 73: 0.166658909313 p/f
Displaying ~1508 seconds to ~1541 seconds. Tracks in blue are outside this range.
Scale: 100px = 2.484m
Comments:
0:25:9-0:25:28 Mouse appears in lower left screen and moves to the center of the lower screen where it disappears under foliage for a little bit but soon reappears (25:24) and heads to the left side of the screen; the Middle mouse heads down screen (towards the new mouse?) but then travels back up screen; the Upper mouse is till stationary by the trap
0:25:29-0:25:33 Mouse in the lower left screen runs along the creek and appears and reappears through the foliage; Middle mouse is going left on the screen; Upper mouse is still stationary by the trap
0:25:34-0:25:34 Mouse in the lower left screen disappears into the foliage
0:25:35-0:25:41 Middle mouse goes up screen then left; Upper mous still stationary by trap
Average Speeds:
No. 73: 0.166658909313 p/f
No. 74: 0.585733011658 p/f
No. 75: 0.236958338336 p/f
No. 76: 0.278560034483 p/f
No. 77: 0.774558728691 p/f
No. 78: 0.809214439541 p/f
No. 79: 0.519821774901 p/f
No. 80: 0.53347575155 p/f
No. 81: 0.338018670657 p/f
No. 82: 0.419493948363 p/f
Displaying ~1541 seconds to ~1558 seconds. Tracks in blue are outside this range.
Scale: 100px = 2.484m
Comments:
0:25:41-0:25:51 Middle mouse disappears; Upper mouse still stationary
0:25:52-0:25:52 Middle mouse reappears
0:25:53-0:25:56 Woodrat enters top right screen; Middle mouse moves a little; Woodrat exits where it entered (does a U-turn); Upper mouse still stationary
0:25:57-0:25:58 Both mice stationary
Average Speeds:
No. 83: 0.387708327646 p/f
Displaying ~1558 seconds to ~1577 seconds. Tracks in blue are outside this range.
Scale: 100px = 2.484m
Comments:
0:25:59-0:26:3 Woodrat reenters; Middle mouse still moving around in its area; woodrat exits where it came in (does a U-turn)
0:26:4-0:26:5 Middle mouse disappears; Upper mous still stationary by trap
0:26:6-0:26:9 New mouse appears on left screen; woodrat re-enters then exits upper right screen
0:26:10-0:26:17 New mouse on the left disappears; Upper mouse still stationary
Average Speeds:
No. 84: 0.383572393649 p/f
No. 85: 0.27397791326 p/f
No. 86: 0.323674081315 p/f
No. 87: 0.57911185573 p/f
Displaying ~1577 seconds to ~1620 seconds. Tracks in blue are outside this range.
Scale: 100px = 2.484m
Comments:
0:26:18-0:26:36 Middle mouse reappears and moves right/up screen
0:26:37-0:26:37 Middle mouse disappears, Upper mouse still stationary by trap
0:26:38-0:26:45 Upper mouse still stationary by trap
0:26:46-0:26:51 Mouse appears on left side of screen by creek, moves down right, then disappears into foliage; Upper mous moving in place a little (chewing/pulling on trap?)
0:26:52-0:26:54 Upper mouse stationary again
0:26:55-0:26:57 Middle mouse reappears in the upper screen and the mouse on the lower left side of the screen reappears as well and moves down screen
0:26:58-0:27:0 Lower left mouse exits bottom screen by creek; Upper and Middle mice move a little
Average Speeds:
No. 88: 0.242281608863 p/f
No. 89: 0.296549050966 p/f
No. 90: 0.171241802376 p/f
No. 91: 0.334232167773 p/f
No. 92: 0.546815424815 p/f
No. 93: 0.587068272277 p/f
No. 94: 0.445476378092 p/f