Implementing a fat based 64 bit dvr
This presentation is the property of its rightful owner.
Sponsored Links
1 / 44

Implementing A FAT Based 64 Bit DVR PowerPoint PPT Presentation


  • 39 Views
  • Uploaded on
  • Presentation posted in: General

Implementing A FAT Based 64 Bit DVR. Peter VanOudenaren, President, EBS Inc. DVR System Requirements. Must minimize latencies Must minimize CPU utilization. Must have predictable execution times Must support contiguous files Must be DMA friendly. DVR System Requirements Continued….

Download Presentation

Implementing A FAT Based 64 Bit DVR

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Implementing a fat based 64 bit dvr

Implementing A FAT Based 64 Bit DVR

Peter VanOudenaren, President, EBS Inc.


Dvr system requirements

DVR System Requirements

  • Must minimize latencies

  • Must minimize CPU utilization.

  • Must have predictable execution times

  • Must support contiguous files

  • Must be DMA friendly


Dvr system requirements continued

DVR System RequirementsContinued…..

  • Must guarantee volume integrity.

  • Must guarantee session integrity.

  • Must monitor resource utilization and block placement


Dvr application requirements

DVR Application Requirements

  • Must support files over 4 Gigabytes

  • Must continuously collect incoming video to a disk buffer.


Dvr application requirements continued

DVR Application RequirementsContinued…..

  • Must simultaneously record and display data from the disk buffer.

  • Must archive data from the disk buffer.

  • Must navigate the disk buffer.


Rtfsproplus achieves dvr system and application requirements

RTFSProPlus Achieves DVR System and Application Requirements

  • Extremely fast command executions speeds

  • Failsafe III maintains volume integrity

  • Failsafe III maintains session integrity

  • Failsafe III extends the “real time” interval


Rtfsproplus achieves dvr system and application requirements1

RTFSProPlus Achieves DVR System and Application Requirements

  • Asynchronous operation

  • FAT64 64 bit metafiles

  • Contiguous files

  • Real time read

  • Real time write

  • Real time seek


Dvr solutions with rtfsproplus

DVR Solutions With RTFSProPlus

  • Remote diagnostics tool

  • Network attached remote debugger

  • Real time performance monitoring

  • Real time monitoring of resource usage

  • Inspect FAT regions and file chains

  • Browse file directories

  • Identify corrupted files

  • Produce quality assurance reports


Dvr solutions with rtfsproplus1

DVR Solutions With RTFSProPlus

  • No copy extracts of linear segments from the circular buffer

  • Circular file writes and reads process extract regions appropriately

  • Extracted regions appear simultaneously in a linear data file and the circular data buffer


Dvr solutions with rtfsproplus2

DVR Solutions With RTFSProPlus

  • Circular (ring buffer) files for continuous collection of incoming video

  • Separate read and write file pointer

  • “Stream view” file pointers

  • “File view” file pointers


Dvr solutions with rtfsproplus3

DVR Solutions With RTFSProPlus

API designed specifically for DVR and high speed data acquisition applications

pc_cfilio_open - Create a circular buffer

pc_cfilio_close- Close a circular buffer

pc_cfilio_read- Read data from circular buffer

pc_cfilio_write- Write to circular buffer

pc_cfilio_lseek- Seek read or write pointer

pc_cstreamio _lseek- Seek read or write stream

pointer

pc_cfilio_extract- Create an archive file from a

section of the circular buffer


Dvr solutions with rtfsproplus4

DVR Solutions With RTFSProPlus

pc_cfilio_open – Open a circular

file.

  • Select file wrap-around point

  • Select 32 bit or 64 bit file

  • Select block allocation policy

  • Provide extract region buffering


Dvr solutions with rtfsproplus5

DVR Solutions With RTFSProPlus

pc_cfilio_read – Read from a

circular file.

  • DMA friendly

  • Guaranteed to access disk only for file data

  • Wraps at file wrap point

  • Stops when read catches write file pointer

  • Reads extract file regions appropriately


Dvr solutions with rtfsproplus6

DVR Solutions With RTFSProPlus

pc_cfilio_write – Write to a

circular file.

  • DMA friendly

  • Guaranteed to access disk only for writing

  • file data

  • Wraps at file wrap point

  • Programmable to stop when it catches read

  • file pointer

  • Overtakes extract file regions appropriately


Dvr solutions with rtfsproplus7

DVR Solutions With RTFSProPlus

pc_cfilio_lseek – Seek in a circular

file.

  • Maintains separate read an write pointers

  • Uses full 64 bit arguments

  • Extremely fast, guaranteed zero disk accesses

  • File pointers represents a linear view of the

  • underlying file

  • Seeks within extract file regions appropriately


Dvr solutions with rtfsproplus8

DVR Solutions With RTFSProPlus

pc_cstreamio_lseek – Circular file

seek.

  • Maintains separate read an write pointers

  • Uses full 64 bit arguments

  • Extremely fast, guaranteed zero disk accesses

  • File pointers represent 64 bit bite offsets in the

  • incoming data stream

  • Seeks within extract file regions appropriately

  • Eases mapping time offsets to video content in

  • the buffer


Dvr solutions with rtfsproplus9

DVR Solutions With RTFSProPlus

pc_cfilio_extract – Extract video from the

buffer to archive files

  • Extract N bytes from the current read file pointer.

  • Real time, requires zero data copying in most cases.

  • Minimal copying for non-cluster aligned extract

  • regions

  • Establishes soft link between the extract file and ring

  • buffer

  • Extracts to and from either FAT64 files or FAT32 files.

  • Allocates clusters to be used when the regions is

  • overwritten


Dvr programming examples

DVR Programming Examples

The following slides present the source code and run time behavior of typical DVR algorithm implemented with RtfsProPlus.

  • Creating a circular file

  • Buffering incoming video

  • Displaying buffered video

  • Accessing frames in the buffer

  • Buffering at steady state

  • Archiving buffered video

  • Displaying archived video


Creating a circular file

Creating a Circular File

int VideoBufferFd;

int DisplayFromFd;

int ArchiveVideoBufferFd;

char *VideoBufferFileName = "dvr.demo";

char *VideoBufferDrive = "A:";

/* Create a circular buffer file that wraps at 16 Gigabytes and keeps writing even if the write pointer overtakes the read pointer */

BOOLEAN InitializeVideoBuffer(dword FrameNumber)

{

dword cluster_size_bytes;

EFILEOPTIONS Options;

rtfs_memset(&Options, 0, sizeof(Options));


Creating a circular file1

Creating a Circular File

Options.allocation_policy = PCE_CIRCULAR_BUFFER;

/* Since the file is over 4 gigabytes, Use FAT64 */

Options.allocation_policy |= PCE_64BIT_META_FILE;

Options.circular_file_size_hi = 4; /* 4 X 4 gigabytes */

Options.circular_file_size_lo = 0; /* Wrap point */

/* Force it to be contiguous and preallocate all clusters*/

Options.allocation_policy |= PCE_FORCE_CONTIGUOUS;

/* How many cluster in 16 Gigabytes */

cluster_size_bytes = pc_cluster_size(VideoBufferDrive);

Options.min_clusters_per_allocation =

16 * (1073741823/cluster_size_bytes);

VideoBufferFd = pc_cfilio_open ((byte *)VideoBufferFileName,

(PO_BINARY|PO_RDWR|

PO_CREAT|PO_TRUNC), &Options);

DisplayFromFd = VideoBufferedFd;

if(VideoBufferedFd >= 0) // Return the Status

return(TRUE);

else

return(FALSE);

}


Buffering incoming video

Buffering Incoming Video

Background thread cycles continuously, waits

for a Framebuffer to be captured then writes to

the circular file and releases the Framebuffer

memory for future capture


Buffering incoming video1

Buffering Incoming Video

void VideoInputThread(void)

{

int FrameSize;

dword FrameAddress, nWritten;

for(;;)

{

WaitForFrame(&FrameSize, &FrameAddress);

pc_cfilio_write(VideoBufferFd, (byte*)FrameAddress,

FrameSize, &nWritten);

ReleaseFrameMemory(FrameAddress, FrameSize);

}

}


Displaying buffered video

Displaying Buffered Video

Background thread cycles continuously,

acquires an available video display page, then

reads data from the circular file into the video

memory and alerts the video processor that the

frame may now be displayed


Displaying buffered video1

Displaying Buffered Video

void DisplayFromVideoBuffer(void)

{

for(;;)

{

VideoPage = GetNextFreeVideoPage(&PageAddress, &PageSize);

ClaimVideoProcessor();

// Display from the circular file unless we are displaying a saved video clip

if (DisplayFromFd == VideoBufferedFd)

pc_cfilio_read (VideoBufferFd, PageAddress, PageSize, &BytesRead);

else

{ // Displaying an archive

pc_efilio_read(DisplayFromFd, PageAddress, PageSize, &BytesRead);

if (!BytesRead)

{ // The clip finished. resume displaying from the video buffer

pc_efilio_close(DisplayFromFd);

DisplayFromFd = VideoBufferedFd;

pc_cfilio_read (VideoBufferFd, PageAddress, PageSize, &BytesRead);

}

}

QueueDisplayVideoPage (VideoPage);

ReleaseVideoProcessor ();

}

}


Accessing frames in the buffer

Accessing Frames In The Buffer

Seek to a specific Frame in the video buffer

where DisplayFromVideoBuffer() will resume

displaying data from.


Accessing frames in the buffer1

Accessing Frames In The Buffer

BOOLEAN SetVideoBufferDisplayLocation(dword FrameNumber)

{

ddword stream_offset;

dword CurrLPtrHi, CurrLPtrLo, newFramePtrHi, newFramePtrLo;

ClaimVideoProcessor();

// save the current display location

if (DisplayFromFd == VideoBufferedFd)

pc_cfilio_lseek (VideoBufferFd, CFREAD_POINTER, 0, 0,

PSEEK_CUR, &CurrLPtrHi, &CurrLPtrLo);

else

pc_efilio_lseek (DisplayFromFd, 0, 0, PSEEK_CUR, &CurrLPtrHi, &CurrLPtrLo);

// Now seek to the byte location of the frame

stream_offset = FrameNumber * BYTES_PER_FRAME;

if (DisplayFromFd == VideoBufferedFd)

SeekOk = pc_cstreamio_lseek (VideoBufferFd, CFREAD_POINTER, M64HIGHDW(stream_offset),

M64LOWDW(stream_offset), PSEEK_SET,

&newFramePtrHi, &newFramePtrLo);

else

SeekOk = pc_efilio_lseek (DisplayFromFd, M64HIGHDW(stream_offset),

M64LOWDW(stream_offset), PSEEK_SET,

&newFramePtrHi, &newFramePtrLo);


Accessing frames in the buffer2

Accessing Frames In The Buffer

if (SeekOk)

{

// The read pointer has been moved to this frame allow the display process

// to resume from here

ReleaseVideoProcessor();

return(TRUE);

}

else

{

// Seek failed make sure the pointer is at the original location

if (DisplayFromFd == VideoBufferedFd)

// This frame number is not buffered in the 16 gigabyte circular

pc_cfilio_lseek (VideoBufferFd, CFREAD_POINTER, &CurrLPtrHi, &CurrLPtrLo, PSEEK_SET, &CurrLPtrHi, &CurrLPtrLo);

else

// This frame number is not buffered in the video clip

pc_efilio_lseek (DisplayFromFd, &CurrLPtrHi, &CurrLPtrLo, PSEEK_SET, &CurrLPtrHi, &CurrLPtrLo);

ReleaseVideoProcessor();

return(FALSE);

}

}


Buffering at steady state

Buffering At Steady State

Demonstrate using circular files for a burst

data collection application with deep buffering

requirements.

Data retention is critical so if the application

loses any data it is a critical error and must be

restarted with more buffering, faster

processing or slower collection rates.


Buffering at steady state1

Buffering At Steady State

void CollectBurstInput()

{

EFILEOPTIONS Options;

dword cluster_size_bytes,nWritten,nRead,nBurstdata;

int CFileFd;

byte *PburstData, Buffer[4096];

rtfs_memset(&Options, 0, sizeof(Options));

/* Create a circular file that wraps at 16Gigabytes and stops writing if the write pointer overtakes the read pointer */

Options.allocation_policy = PCE_CIRCULAR_FILE;

/* Since the file is over 4 gigabytes, Use FAT64 */

Options.allocation_policy |= PCE_64BIT_META_FILE;

Options.circular_file_size_hi = 4; /* 4 X 4 gigabytes */

Options.circular_file_size_lo = 0; /* Wrap point */

/* Force it to be contiguous and preallocate all clusters*/

Options.allocation_policy |= PCE_FORCE_CONTIGUOUS;

/* Make it a temp file since there's no need to ever flush the file's metadata to disk

Options.allocation_policy != PCE_TEMP_FILE;

/* How many cluster in 16 Gigabytes */

cluster_size_bytes = pc_cluster_size(DataBufferDrive);

Options.min_clusters_per_allocation = 16 * (1073741823/cluster_size_bytes);

CFileFd = pc_cfilio_open ("DataCollectBufferFIle", PO_BINARY|PO_RDWR|PO_CREAT|PO_TRUNC, &Options);

if(CFileFd < 0)

return;


Buffering at steady state2

Buffering At Steady State

/*Enter a loop, accepting variable amount of data from a fictitious data collection routine and put the data in a ring buffer.

Read the data in 4096 byte chunks from the ring buffer and pass it to a fictitious processing routine.

If the processing loop empties the buffer, wait for more data.

If the write operation can't write all of the data that the data collection routine provides that indicates it would overwrite data that has not yet been collected, so warn the user and halt the process*/

for (;;)

{

if (BurstDataAvailable(&PburstData, &nBurstdata)

{

pc_cfilio_write(CFileFd, PburstData, nBurstdata, &nWritten);

if (nWritten != nBurstdata)

{

printf("File buffer is too small, lost data\n");

pc_cfilio_close(CFileFd);

return;

}

}

pc_cfilio_read(CFileFd, Buffer, 4096, &nRead);


Buffering at steady state3

Buffering At Steady State

if (nRead)

ProcessData(Buffer,nRead);

else

WaitForMoreData();

}

}


Archiving buffered video

Archiving Buffered Video

Extract some number of frames from the video

buffer to an archive file. The extracted data will

be stored in a linear file, but it will also be part

of the video buffer file.


Archiving buffered video1

Archiving Buffered Video

/* Callback function that closes the archive file when it is no longer needed by the circular file. This occurs when the write pointer reaches and overtakes the region in the buffer where the data was extracted from. */

void CBArchiveOverwritten(int fd, BOOLEAN Success)

{

//Start an asynchronous close, the background async handler will complete it for us

pc_efilio_async_close(fd);

}

BOOLEAN ArchiveFromVideoBuffer(char *ArchiveFileName, dword StartFrame,

dword nFrames, BOOLEAN DoFlush)

{

int ExtractFd;

ddword stream_offset,extract_bytes;

EFILEOPTIONS ArchiveFileOptions;

BOOLEAN Result = FALSE;

/* open a special REMAP file for archiving */

rtfs_memset(&ArchiveFileOptions, 0, sizeof(ArchiveFileOptions));

ArchiveFileOptions.allocation_policy = PCE_REMAP_FILE;

ArchiveFileOptions.allocation_policy |= PCE_64BIT_META_FILE;

ExtractFd = pc_efilio_open(ArchiveFileName, PO_RDWR|PO_TRUNC|PO_CREAT,

PS_IWRITE | PS_IREAD, &ArchiveFileOptions);

if (ExtractFd < 0)

return(FALSE);


Archiving buffered video2

Archiving Buffered Video

// This moves the read pointer but happens very quickly, so wait for retrace

WaitForVideoRetrace();

ClaimVideoProcessor();

//save current display location

pc_cfilio_lseek (VideoBufferFd, CFREAD_POINTER, 0, 0, PSEEK_CUR,

&CurrLPtrHi, &CurrLPtrLo);

// Now seek to the byte location of the frames to archive

stream_offset = StartFrame * BYTES_PER_FRAME;

if(pc_cstreamio_lseek (VideoBufferFd, CFREAD_POINTER, M64HIGHDW(stream_offset), M64LOWDW(stream_offset), PSEEK_SET, &newFramePtrHi, &newFramePtrLo))

{

extract_bytes = nFrames * BYTES_PER_FRAME;

/* link the two files together with little or no copying */

Result = pc_cfilio_extract(VideoBufferFd, ExtractFd, M64HIGHDW(extract_bytes), M64HIGHDW(extract_bytes));

}


Archiving buffered video3

Archiving Buffered Video

// restore the pointer to the original location and resume display

pc_cfilio_lseek (VideoBufferFd, CFREAD_POINTER, &CurrLPtrHi, &CurrLPtrLo, PSEEK_CUR, &newFramePtrHi, &newFramePtrLo);

ReleaseVideoProcessor();

// If success, start an async flush process to put it to disk

if (Result)

pc_efilio_async_flush(ExtractFd);

else

{

// Otherwise delete the empty extract file

pc_efilio_close(ExtractFd);

pc_efilio_async_delete(ArchiveFileName);

}

return (Result);

}


Displaying archived video

Displaying Archived Video

Open a saved video clip for display by the

display thread.


Displaying archived video1

Displaying Archived Video

int CurrentDisplayngArchiveFd;

void DisplayFromVideoArchive(char *ArchiveFileName)

{

int ArchiveFd;

EFILEOPTIONS ArchiveFileOptions;

// Reopen the archive file, the file will automatically detect FAT64 or FAT32

rtfs_memset(&ArchiveFileOptions, 0, sizeof(ArchiveFileOptions));

ArchiveFd = pc_efilio_open(ArchiveFileName, PO_RDWR,0, &ArchiveFileOptions);

if (ArchiveFd >= 0)

{ // Set the file descriptor for the display task

ClaimVideoProcessor();

DisplayFromFd = ArchiveFd;

ReleaseVideoProcessor();

}

}


Completing asynchronous requests

Completing Asynchronous Requests

This background thread cycles continuously and

waits for a time window when it can safely

access the drive without affecting foreground

operations.

It then performs one or more cycles of the

asynchronous processing.


Completing asynchronous requests1

Completing Asynchronous Requests

void AsynchronousCompletionThread(void)

{

int AsyncState, StepPerPass, DriveNumber;

dword FrameAddress, nWritten;

DriveNumber = GetVideoDriveNumber();

for (;;)

{

// Steal cycles for asynchronous processing during a retrace

WaitForVideoRetrace();

StepsPerPass = 1;

/* Do asynchronous processing, on Files, buffers, journal flushes and journal to

volume restores. We could also choose to process only up to an earlier state, such as Journal flush completion by specifying DRV_ASYNC_DONE_JOURNALFLUSH. We could then in a less time critical period complete the whole process. */

AsyncState = pc_async_continue(DriveNumber, DRV_ASYNC_DONE_RESTORE, 1);


Completing asynchronous requests2

Completing Asynchronous Requests

if (AsyncState == PC_ASYNC_COMPLETE)

{

// All done, set a fictitious VolumeUptoDate variable needs to be done

VolumeUptoDate = TRUE;

}

else if (AsyncState = PC_ASYNC_CONTINUE)

{

/* We know that we'll only perform on single multi-block read or one single multi-block write, and we know we can do this in less than 10 miliseconds, so if we have more than 10 miliseconds left take another pass */

if (TimeLeftInRetrace() > 10)

AsyncState = pc_async_continue (DriveNumber,

DRV_ASYNC_DONE_RESTORE,1);

}

if (AsyncState = PC_ASYNC_ERROR)

{

printf ("Error in processing, halt...\n");

halt ();

}

}

}


Rtfsproplus porting

RTFSProPlus Porting


Rtfsproplus ram footprint

RTFSProPlus RAM Footprint


Rtfsproplus code size

RTFSProPlus Code Size

220K (approx) with all features enabled, non-optimized, full debug support


  • Login