Graphics 2 sound
Download
1 / 38

Graphics 2 & Sound - PowerPoint PPT Presentation


  • 274 Views
  • Uploaded on

Graphics 2 & Sound The First Rule of Computer Graphics is: Compress Last Take a picture or scan an image at the highest possible resolution. Work on that image at that resolution. Save the image in its original and compressed formats.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Graphics 2 & Sound' - issac


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Graphics 2 sound l.jpg

Graphics 2 & Sound

CSE5900 -- Intro to MM Computing -- Lecture 3


The first rule of computer graphics is l.jpg
The First Rule of Computer Graphics is:

  • Compress Last

  • Take a picture or scan an image at the highest possible resolution.

  • Work on that image at that resolution.

  • Save the image in its original and compressed formats.

  • Contemporary scanners = 1,200 x 1,200 dots per inch; 32 bit (4 byte) colour, so we have

  • 1200 * 1200 *4 * (say) 7 * 5 = 201,000,000 bytes

  • Hummm. We’re going to need LOTS of RAM. (And hard disk space)

  • (This is why digital cameras are still ‘low resolution’, even with, say, 4,000,000 pixels per image.)

CSE5900 -- Intro to MM Computing -- Lecture 3


Oh dear l.jpg
Oh, Dear

  • If we are working with 200mb files, we have the following problems.

    • We can’t actually see the detail without zooming in a lot.

    • It will take a fair amount of time to get information on to and off of the hard disk.

    • It will take time to get the image to and from:

      • Disk and RAM (video RAM or standard RAM)

      • RAM and CPU

      • CPU to video card

      • Video card to monitor

    • Much of the contemporary evolution in computing tries to improve the bandwidth of these connections

CSE5900 -- Intro to MM Computing -- Lecture 3


Crucial definitions l.jpg
Crucial Definitions

  • Bandwidth

    • The rate at which data moves from point A to point b. The throughput of a connection, measured in either bits or bytes.

  • Bus

    • The physical connection(s) between point A and point B

  • Busses come in two flavours

    • Serial - one signal wire, so 1 bit at a time.

      • COM ports

      • USB (universal serial bus)

    • Parallel - many signal wires (8, 16, 34 or 64 at present), many bits at the same time.

      • PCI bus

      • Northbridge or front end bus

CSE5900 -- Intro to MM Computing -- Lecture 3


Busses words and addresses l.jpg
Busses, Words, and Addresses

  • We can speak of the capacity of four things in terms of the numbers we are discussing:

  • Bus capacity, bits or wires

  • Word length: the number of bits a CPU deals with as a unit. Currently usually 64 bits, and moving towards 128.

  • Maximum address size: the highest location address (for RAM, hard disk, etc) measured in bits.

  • Operating system: The size, in bits, of an instruction. Note that the word size has to be a multiple of the instruction size

CSE5900 -- Intro to MM Computing -- Lecture 3


In the beginning 1983 l.jpg
In the Beginning (1983)

RAM

CPU

Key- board

Floppy

Printer

Screen

CSE5900 -- Intro to MM Computing -- Lecture 3


A little later 1985 l.jpg
A Little Later (1985)

RAM

Video Card

Video

RAM

ALU - Floating

point

Screen

CPU

Key- board

Floppy

Hard

Disk

CSE5900 -- Intro to MM Computing -- Lecture 3


Another view l.jpg
Another View

Mouse

Modem

Printer

LPT2

COM1

Floppy

Hard Disk

Screen

COM2

LPT1

CPU

RAM

IO Card

Video

Card

Disk

Controller

Bus

CSE5900 -- Intro to MM Computing -- Lecture 3


Windows 2 arrives 1992 l.jpg
Windows 2 Arrives (1992)

RAM

ALU - Floating

point

CPU

Key- board

Floppy

Video

RAM

Mouse

Hard

Disk

Video Card

Screen

CSE5900 -- Intro to MM Computing -- Lecture 3


Yesterday l.jpg
Yesterday

CPU

Video

RAM

Video

RAM

Video Card

Games--3D

Video Card

Windows--2D

Screen

CSE5900 -- Intro to MM Computing -- Lecture 3


Today l.jpg
Today

  • RAM

CPU

Video

RAM

Video Card

Windows--2D

Games--3D

Screen

CSE5900 -- Intro to MM Computing -- Lecture 3


Today a second perspective l.jpg
Today - A Second Perspective

Video

RAM

Video Card

RAM

North Bridge

CPU

Cache Memory

South Bridge

Disk

Controller

Sound Card

Other PCI

Cards

CSE5900 -- Intro to MM Computing -- Lecture 3


Tomorrow l.jpg
Tomorrow

Video

RAM

Video Card

Rendering

Physics, etc.

RAM

North Bridge

Cache Memory

CPU

Cache Memory

CPU

South Bridge

Disk

Controller

Sound Card

Other PCI

Cards

CSE5900 -- Intro to MM Computing -- Lecture 3


Ram to screen l.jpg
RAM to Screen

Video RAM

(Frame Buffer)

Screen Pixels

  • A pixel has a matching chunk of video RAM

CSE5900 -- Intro to MM Computing -- Lecture 3


The conversion step l.jpg
The Conversion Step

  • But RAM is digital and video is analog, so we need a digital to analog converter (DAC)

Video

(composite)

RAM (rgb)

RAM

DAC

CSE5900 -- Intro to MM Computing -- Lecture 3


Colour information storage l.jpg
Colour Information Storage

  • 0 to 255 for red 8 bits

  • 0 to 255 for blue 8 bits

  • 0 to 255 for green 8 bits

  • 0 to 255 for Alpha 8 bits

  • So 32 bits (4 bytes) per pixel

  • 1,024 x 768 x 4 = 3,145,728 bytes per frame in video memory

  • So 16mb video memory may be ok, where the image is “pre-cooked” (my jargon)

  • Assuming there’s enough bandwidth for 24 frames/second

CSE5900 -- Intro to MM Computing -- Lecture 3


Pre cooked versus home cooked l.jpg
Pre-Cooked versus Home Cooked

  • Pre-Cooked

    • All of the images are prepared in advance, and are on disk or CD-ROM

    • The processing steps are:

      • Read from hard disk into CPU

      • Decompress compressed images

      • Build the frame in video RAM

      • Run through RAMDAC onto screen

    • The limits are:

      • The bandwidths, especially for video

      • The file sizes versus storage capacities

      • The complexity of decompression

      • Everything has to be completely prepared in advance. It’s all finished art

CSE5900 -- Intro to MM Computing -- Lecture 3


Pre cooked versus home cooked 2 l.jpg
Pre-Cooked versus Home Cooked, 2

  • Home Cooked

    • Where you can’t prepare all images in advance

    • First person perspective games, for example

      • Player can move in any direction, look up, down and around, explore

      • The display has to be created on the fly

      • Curse you, Doom, for starting it all!

  • Pre-cooked is sometimes called “2D” and home cooked is sometimes called “3D”, which is completely misleading.

  • The difference is what’s done in advance or in real time

  • Traditional video cards could do interactive 2D (and pre-cooked 3D)

  • 3D accelerator cards can do interactive 3D image creation in real time.

CSE5900 -- Intro to MM Computing -- Lecture 3


Digression 1 apis l.jpg
Digression 1: APIs

  • API = application programmer’s interface

  • A large set of functions used to extend a standard programming language into a specialized area

  • These functions can be used on the fly as needed

  • There are 2 standard 3D graphics APIs

    • OpenGL (originally from Silicon Graphics, now non-proprietary, used for Doom)

    • Direct3D (from Microsoft, part of DirectX)

  • Allow work (by games programmers) at

    • Low level -- set up scene using graphic primitives

    • High level -- make small variations to a scene -- think of sprites and larger elements

CSE5900 -- Intro to MM Computing -- Lecture 3


Digression 2 building a 3d image l.jpg
Digression 2: Building a 3D Image

  • We have to come up with a 2D screen rendition of a 3D thing which we’ve built on a 2D screen-based workspace

  • First comes the wire frame models of the object. This is all that’s really there, and the rest is just processing.

  • With the pre-cooked, all the processing takes place in advance and the results (not the wire frames) are distributed.

  • With home cooking, the wire frame models are distributed, together with textures and materials, and the processing takes place in real time in the 3D accelerator usually using a game engine created using a API (Direct3D or OpenGL)

CSE5900 -- Intro to MM Computing -- Lecture 3


Digression 2 building a 3d image 2 l.jpg
Digression 2: Building a 3D Image, 2

  • There has to be a camera position from which we look at the wire frames.

  • We have to get rid of things behind other things (clipping)

  • We have to provide light sources, with colour mixes, intensities and directions, resulting in shadows, reflections, etc. (ray tracing, usually faked in games)

  • We have to apply different textures (e.g., red gum) to the surfaces of the wire frames (texture mapping)

  • We have to smooth everything out according to all the above (rendering)

  • The 3D accelerator cards provide the tools to do this kind of stuff in real time

CSE5900 -- Intro to MM Computing -- Lecture 3


How well do they work l.jpg
How Well Do They Work?

  • Consider that each frame in an animated film takes a network of computers many minutes to render.

  • Consider that the wire frames for animated films have hundreds of thousands or millions of polygons (the primitive used for processing)

  • Consider that the audience for 3D accelerators is 16 year old boys who haven’t graduated from killing things (a low resolution occupation) to sex (a high resolution preoccupation)

  • The cards are getting good at keeping their intended audience happy

  • But they are miles from being able to do Riven in real time

CSE5900 -- Intro to MM Computing -- Lecture 3


Music hath charms to sooth the savage beast l.jpg

Music Hath Charms to Sooth the Savage Beast

Introduction to Sound Processing

CSE5900 -- Intro to MM Computing -- Lecture 3


Sound is analog l.jpg
Sound Is Analog

  • So there’s infinite variation

  • Like a rock thrown into a pond, there are waves:

    • Amplitude: how high the waves are -- Loudness

    • Frequency: how many waves per second -- Pitch

  • Loudness is measured in decibels

    • This is a log scale, so 20 is ten times as loud as 10, 30 is ten times as loud as 20, and so forth.

    • You can distinguish from just over 0 dB to 120dB

      • 37 quiet office (no air-conditioning)

      • 59 conversation

      • 76 loud factory

      • 110 really loud night club or rave

      • 140 threshold of pain (well, for some)

CSE5900 -- Intro to MM Computing -- Lecture 3


Sound is analog 2 l.jpg
Sound Is Analog, 2

  • Pitch is measured in Hz (Hertz, cycles per second) and kHz.

  • You can distinguish between a few Hz and 15 - 20 khz (this is age dependent)

    • Lowest note on piano 27 Hz

    • Highest note on piano 4,186 Hz (4.186 kHz)

    • Lowest vocal sound 80 Hz

    • Highest vocal sound 800 Hz

    • The A above middle C 440 Hz (used to be lower!)

  • But a sine wave just at these frequencies sounds sterile: it lacks the overtones, the harmonics, produced by all natural sources of sound. Think of the FM synthesis sound of a cheap sound card.

CSE5900 -- Intro to MM Computing -- Lecture 3


Sound is analog 3 l.jpg
Sound Is Analog, 3

  • An instrument vibrating produces lots of sounds above the fundamental tone.

  • Many of these are various octaves above the fundamental

    • Octave = double the frequency

    • To get realistic sound we have to pick up at least the 4th harmonic, 4x the frequency of the fundamental.

    • So we have to pick up to 12kHz for, say a realistic flute sound (where the highest fundamental is just under 4kHz

    • More is better until, say 20kHz where a 5 year old’s hearing cuts out.

CSE5900 -- Intro to MM Computing -- Lecture 3


Sound into bits adc l.jpg
Sound Into Bits (ADC)

  • Something that always confuses me:

    • The 16 bits used (0 - 64k) record the amplitude (loudness)

    • The differences between successive 16 bit samples contain the frequency (pitch)

    • Remember, a high wave also has a trough between peaks.

  • So how often do we have to sample to get enough samples

    • 2x the maximum difference we want to catch.

    • And we want to catch differences up to 20mHz, so we have to sample at at least 40,000 times a second.

    • So your CDs contain music sampled at just over 44,000 times a second.

    • So the digital signal bandwidth must be 16 x 44,000 = 704,000 bits per second or 88 kbps. With stereo sound, we have to have two such samples, so a 1x CD-ROM bus goes at 176 kbps, which we already knew!

CSE5900 -- Intro to MM Computing -- Lecture 3


Bits into sound dac l.jpg
Bits Into Sound (DAC)

  • Amplifiers and speakers are analog devices, so

  • The CD player does DAC and passes the results as an analog signal to your stereo system.

  • It does the same if you listen to music off your CD-ROM drive.

  • But where does your sound card do the conversion?

  • Hummmm…. Later is (far, far) better, because there’s lots of electrical interference inside your PC. Digital isn’t affected by this, but analog is.

  • So the perfect system would be all digital inside the computer and have its DAC inside the speakers

  • The USB promises to make this real, and Microsoft makes speakers which work this way.

CSE5900 -- Intro to MM Computing -- Lecture 3


Midi general midi l.jpg
MIDI & General MIDI

  • MIDI = Music Instrument Digital Interface

  • MIDI is to sampling exactly what vector is to raster graphics

    • A language for describing sounds

      • The notes

      • The instruments, each of which has a number

        • 128 instruments

        • Plus drum kit

      • The note characteristics

        • attack

        • sustain

        • decay

        • release

      • 2+ ways of making those notes

        • FM synthesis

        • Wavetable

CSE5900 -- Intro to MM Computing -- Lecture 3


The parts of a midi note l.jpg
The Parts of a MIDI Note

  • From the MIDI Manufacturers Homepage

CSE5900 -- Intro to MM Computing -- Lecture 3


Making midi l.jpg
Making MIDI

  • FM Synthesis

    • Sterile sine waves

    • What gave computer music a bad name

  • Wavetable Sound Generation

    • The music gives the number of the instrument

    • Samples of the sound of that instrument are stored in ROM/RAM on the sound card

    • The samples are processed to give a far better illusion of the sound of the instrument

    • The more samples, the better, so 4mb of samples on ROM are better than 512k.

    • Wavetables may also be downloaded from CD-ROMS

    • Wavetables can be purchased as daughterboards for better sound cards.

CSE5900 -- Intro to MM Computing -- Lecture 3


Midi quality l.jpg
MIDI Quality

  • Well, as always there’s the trade off:

    • Much smaller file size

    • Always somewhat less quality

    • Infinitely cheaper to create -- only one muso necessary

    • May require significant CPU processing

CSE5900 -- Intro to MM Computing -- Lecture 3


Channels voices and streams l.jpg
Channels, Voices and Streams

  • A channel drives a speaker:

    • 2 channels for standard stereo

    • 4 channels for 3D sound (two may be faked)

    • 8+ channels for super sound in theatre movies

  • A voice is an instrument, etc. on a channel

    • MIDI supports a large number of voices: 32, 64. This is polyphony

    • The voices are superimposed, in digital or analog form, and then sent to the speakers

    • Again, multiple voices may load down the CPU

  • A stream is half voice and half channel

    • Lets record a sound effect, a stream

    • When we need it, we superimpose it on top of the sound going to a channel

    • The sound card and/or CPU do the work

CSE5900 -- Intro to MM Computing -- Lecture 3


Channels voices and streams 2 l.jpg
Channels, Voices and Streams, 2

  • The higher the bandwidth into the sound card

    • The more channels, voices and streams we can get at once

    • And the more processing work has to be done

    • So we either do more on-sound-card processing or bog down the CPU

    • (Sound like the issues related to 3D accelerator cards?!)

  • The 16 bit ISA bus worked fine for sound until we decided to want more channels, voices, streams, wavetables, etc. (But they still flourish: somebody tried to sell me one yesterday)

  • Newer cards use the PCI bus, and make use of it.

  • And use a newer generation of sound chips.

  • But the evolution here is relatively slow!

CSE5900 -- Intro to MM Computing -- Lecture 3


Games and computer sound l.jpg
Games and Computer Sound

  • Games are one of several factors driving the evolution of graphics boards

  • Games are almost the only factor driving the evolution of sound cards

    • Who is sneaking up behind me? We need 3D.

    • What kind of sound does that alien make when exploded? We need lots of streams superimposed.

  • 3D illusion

    • Uses 3 speakers (woofer, + 2 satellite) and an algorithm to fox the ear by marginally delaying one stereo channel

    • Developed by NASA for space flight simulators

    • Can work well if don’t move your head

    • With 5 speakers, esp. with 4 channels, can work very well indeed

  • As the musicians are never behind you, not necessary for music. Whoops, sorry Berlioz, Allegri, Tchkovsky, etc.

CSE5900 -- Intro to MM Computing -- Lecture 3


Games and computer sound 2 l.jpg
Games and Computer Sound, 2

  • Competing 3D positional audio standards

    • A3D

      • From Aureal Semiconductor (now defunct)

      • Now a non-runner

    • Audio Extensions .EAX

      • From Creative Labs (who brought us Sound Blaster)

      • Bought resources of Aureal (A3D)

      • Included A3D technology in .EAX

      • In Nomad, Zen and other sound card brands

    • DirectSound3D

      • From Microsoft

      • Part of the DirectX set of Windows APIs/extensions, including Direct3D

    • CRL Sensaura

      • Bought by Creative Technology in 2003

      • Used in Xbox, PlayStation2, Nintendo GameCube, PCs

CSE5900 -- Intro to MM Computing -- Lecture 3


Digital signal processing l.jpg
Digital Signal Processing

  • Now most music processing is in analog form on card.

  • Far better if it were all digital, with DAC and amplification on the speakers

  • Need DAE: Digital Audio Extraction, from CD/CD-ROM

  • Introduced with 24x CD-ROMs.

  • Theory is that we use USB to get the digital sound to the speakers

  • Speakers with USB connections now also have standard audio connections for the sounds and use USB for controlling the settings

  • Amplifiers are both on the sound card and external.

CSE5900 -- Intro to MM Computing -- Lecture 3


Slide38 l.jpg
MP3

  • MPEG-1, Layer 3 sound compression -- intended for movies on CD and DVD

    • 90+% compression possible

    • Compression takes about a third of the length of the music

    • A typical song (50mb) goes to 5mb

    • Is a lossy compression, so the quality goes down, but not much

    • No encryption in any way

    • No “watermark” (watermark = a secret pattern of bits somewhere which indicates the source of the copy)

    • Much music publisher panic with the popularity of the format.

    • Much more music publisher panic with the IPod, which can store an entire music library in MP3 formt

CSE5900 -- Intro to MM Computing -- Lecture 3