Training course on edit
This presentation is the property of its rightful owner.
Sponsored Links
1 / 60

Training Course on EDIT PowerPoint PPT Presentation


  • 168 Views
  • Uploaded on
  • Presentation posted in: General

Training Course on EDIT. For Users. Outline of the module. Introduction Using EDIT - integration with other tools Objects in EDIT for Users EDIT Graphical User Interface Future developments. A - Introduction. EDIT is a tool for data validation - data edit/imputation.

Download Presentation

Training Course on EDIT

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Training course on edit

Training Course on EDIT

For Users


Outline of the module

Outline of the module

  • Introduction

  • Using EDIT - integration with other tools

  • Objects in EDIT for Users

  • EDIT Graphical User Interface

  • Future developments


A introduction

A - Introduction


Edit is a tool for data validation data edit imputation

EDIT is a tool for data validation - data edit/imputation

  • What is data validation?- An activity aimed at verifying whether the value of a data item comes from the given set of acceptable values:

  • What is data editing? - The activity aimed at identifying erroneous entries and correcting them if necessary. Example: the response is missing or incorrect.


How edit works shortly

How EDIT works shortly?

A format contains a description of the data in a dataset

A dataset is a set of data according to a specific format

Define a format

Define a program containing rules and file operations to be executed on the dataset(s)

Uploads dataset(s) from external files

For users

Execute the job

Get the report containing errors (if any)


Edit user types

EDIT User types

  • 'User‘ - Executes programs on datasets and accesses the reports.

  • 'Programmer‘ - Manages the metadata needed by the user to execute programs;

    • Implements 'formats‘;

    • Implements ‘validation rules’ by means of 'programs';

    • Defines other operations on files by mean of 'programs';

    • Sets up the unattended mode configuration.

  • 'Administrator'

    • Manages users and permissions.


User type functionalities

'User' type functionalities

  • ‘Change Password’

    • Allows users to change their password;

  • ‘Dataset Import/Export’

    • Allows users to import and export data to and from EDIT as well as monitor any ongoing import/export processes;

  • ‘Job Execution’

    • Allows users to execute programs on imported datasets and view/export the results of the execution.


The user workflow

The 'User' Workflow


The link between user workflow and user interface

The link between 'User workflow' and 'User interface'


What can we do by means of a program

What can we do by means of a ‘program’?

  • Run programs containing mainly validation rules / computations:

    A1 – Single column – only a column is involved;

    A2 – Multiple columns – two or more columns within a single record are involved;

    B - Vertical – multiple records involved;

    C - Hierarchical – multiple datasets involved.

  • Perform dataset operations: Copy, Merge, Alter, Aggregate, etc.

  • Use specialised functions like outlier detection: Terror, Hidiroglu-Berthelot, σ-Gap;

  • Accepted formats: SDMX-ML, GESMES, CSV, FLR.


Accepted data formats

Accepted data formats

CSV (with or without header) (SBS, CVTS,TOURISM)

9H; 2008; LT; 2; B-N_X_K642; 11930; 16236; ; ; ; ; UNIT; ; ; ; ; ; TT0; ; ; ; ; D08

9H; 2008; LT; 3; B-N_X_K642; 11930; 1001; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08

9H; 2008; LT; 4; B-N_X_K642; 11930; 529; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08

9H; 2008; LT; 30; B-N_X_K642; 11930; 17766; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08

9H; 2008; LT; 2; B-E; 11930; 1138; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08

9H; 2008; LT; 3; B-E; 11930; 104; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08

9H; 2008; LT; 4; B-E; 11930; 61; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08

multi-year 2007, 2008, 2009 observations

FLR example 1

001E20100121814 00 804.822

001E20100121816 93 5295.54

001E20100121814 99 6166.24

001E20100125290334 581.371

FLR example 2

2010010011 010252000405595911005909580E 01ZZZZZ 2691.966 2734482.0 0.0

2010010011 010252000405595911004009600E 01ZZZZZ 237.543 341202.0 0.0

GESMES(BOP ITS, BOP FDI)

UNA:+.? '

UNB+UNOC:3+FR2+4D0+100929:1637+IREF000243++GESMES/TS'

UNH+MREF000001+GESMES:2:1:E6'

BGM+74'

NAD+Z02+ECB'

NAD+MR+4D0'

NAD+MS+FR2'

IDE+10+EUROSTAT_BOP_01 reporting'

DSI+BOP_FDI_A'

STS+3+7'

DTM+242:201009291637:203'

DTM+Z02:20072009:702'

IDE+5+EUROSTAT_BOP_01'

GIS+AR3'

GIS+1:::-'

ARR++A:FR:N:2:330:N:4A:E:9999:9999:20072009:702:0:A:F+0:A:F+0:A:F‘

ARR++A:FR:N:2:330:N:4F:E:9999:9999:20072009:702:0:A:F+0:A:F+0:A:F'

ARR++A:FR:N:2:330:N:7Z:E:9999:9999:20072009:702:0:A:F+0:A:F+0:A:F'

ARR++A:FR:N:2:330:N:A1:E:1100:9999:20072009:702:5824:A:F+5930:A:F+4204:A:F'

ARR++A:FR:N:2:330:N:A1:E:1495:9999:20072009:702:5828:A:F+5932:A:F+4206:A:F'


B using edit integration with other tools

B - Using EDIT - integration with other tools


Ways of using edit

Ways of using EDIT

  • As a web-based application – called by other applications;

  • Standalone – running on a PC;

  • Client – server – running in a Data Centre.


Edit as web based application

EDIT as Web-based application

  • Web-based Interface

    • Unified interface for both the standalone version and the server deployment;

    • EUROSTAT Look & Feel;

    • Light interface, simplified workflows.

  • ECAS account is needed.


Edit running standalone

EDIT running standalone

  • Downloadable package;

  • Standalone installation supported by Windows XP and Windows 7;

  • Simple installation wizard;

  • Full functionality;

  • Standard authentication is requested.


Client server mode for edit

Client - server mode for EDIT

  • EDIT runs on a UNIX machine;

  • The current setup is EDIT installed at Eurostat & other DGs;

  • Contains all registered domains (= user specific workspaces) as by default imbedded;

  • ECAS credentials needed for external users.


Edamis integration

EDAMIS integration

  • EDAMIS allows transmitting data files through a single entry point;

  • EDAMIS can send data to EDIT by placing the files in a configurable location;

  • EDIT detects metadata based on the EDAMIS naming convention;

  • EDIT performs the processing in unattended mode.


Sdmx integration

SDMX integration

  • Statistical Data and Metadata Exchange (SDMX) initiative is sponsored by seven institutions (the BIS, the ECB, Eurostat, the IMF, the OECD, the UN and the World Bank);

  • SDMX describes and universalises the way to exchange statistical data and metadata;

  • EDIT can import SDMX-ML datasets.


C objects in edit for users

C - Objects in EDIT for Users

  • Datasets instantiations - lookups;

  • Programs, jobs


1 dataset instantiations

1 - Dataset instantiations

  • Dataset Instance (Dataset) – a collection of data rows according to the structure of a format;

  • A two dimensional table composed by rows and columns:

    • Columns correspond to the fields defined in the format;

    • Records – no limit on size or number.


Dataset example table aes adult education survey

Dataset example – Table AES (Adult Education Survey)


The description of the table aes

The description of the table AES


Example format dataset instantiation

Example: 'Format' – 'Dataset instantiation'

Format

Dataset instantiation


The same format different datasets

The same format – different datasets


Lookup tables code lists

Lookup tables – code lists

  • Lookup – An auxiliary dataset containing a list of values to be used for validating codes;

  • Code lists – usually lookup tables refer to code lists;

  • One can use several code lists inside the same program – as many as needed for the given data sets – 'Country', NACE, NUTS;

  • Several versions of the same code list can be used from within the same program, if needed.


2 programs jobs

2 - Programs, jobs

  • Program – a set of operations to be performed on a specified dataset definition (format);

  • No specific dataset is associated with a program, only formats (dataset definitions) should be specified;

  • Job – the association between a 'Program' and concrete 'Dataset Instances';

  • Possible operations types of rules/checks: Single and Multiple column(s), Vertical and Hierarchical.


Validation report

Validation report

  • It contains:

    • Job results – information about the job;

    • Error statistics – summary of the errors;

    • Error report – detailed list of errors.


Error statistics

Error statistics

  • The error statistics are displayed in a table format and it consists of the following columns:

    • Rule name: The name of the program rule that failed;

    • No of Failures: Individual rows that the error appeared through job execution;

    • Rule Message: Rule’s error message as defined in the program.


Errors statistics

Errors statistics


Detailed error report

Detailed error report


D edit graphical user interface

D - EDIT GRAPHICAL USER INTERFACE


Edit log in

EDIT - Log in


Edit home page

EDIT Home page

Menu options

User profile information

Herepasswordcanbechanged


Defining dataset import dataset

Go in >Dataset>> Import dataset

Screen part I

Defining dataset: import dataset

Select a file on your hard drive

Select a file type (CSV / GESMES / FLR / SDMX)

Reusesavedparameters

Starting line

Save properties for further use


Defining dataset import dataset1

Screen part II

Select a format

Defining dataset: import dataset

Reusesaved configuration

Select columns to import

Use the arrows to addremovefields

Provide a name for the new dataset

Save configuration for further use

Click to import


Defining dataset import dataset2

Unsuccessful import

Defining dataset: import dataset

Click to download the importing report in text format

Statusis FAILED


Defining dataset import dataset3

Successful import with warnings

Defining dataset: import dataset

In the report, two records wereskipped (lines 2 and 5)

Click to download the importing report in text format

Statusis COMPLETED


Defining dataset import dataset4

Successful import

Defining dataset: import dataset

Afterimporting, EDIT redirectsyou to the searchdatasetscreen

Click to look at the content imported

Delete a selecteddataset

Statusis COMPLETED


Defining dataset import dataset5

Defining dataset: import dataset

Click to hiddenfileds

Select fields to behidden in the display

Hiddenfields

EDIT hides the selectedfields


Defining dataset import dataset6

Defining dataset: import dataset

Unfold the Basic filtering options

Select an logicaloperator

Enter a value

Select a field in the datatset (e.g. WEIGHT)

The corresponding records are filtered


Defining dataset import dataset7

Defining dataset: import dataset

Unfold the Advanced filtering options

Create an expression aided by the lists of fields, operators and functions

Click to apply the searchcriteria

The corresponding records are filtered


Defining dataset import dataset8

Defining dataset: import dataset

Customizeyourview

Export in CSV format


Defining dataset search dataset

Defining dataset: search dataset

Searchcriteria

Restore an archiveddataset

Export the dataset in CSV format

List of alreadyimporteddatasets

Viewdetails of the datasetwithfiltering options

Delete the dataset

Archive the dataset


Defining dataset import export dataset

Defining dataset: Import/Export dataset

Import/Export historysearch

Searchcriteria

Viewdetails of the datasetwithfiltering options

List of Import/export history

Delete the dataset


Defining jobs create a job

Defining jobs: Create a job

Menu option

Searchcriteria

Click to create a job for this program

List of existing programs to beexecuted


Defining jobs create a job1

Defining jobs: Create a job

Enter a name and a description

Choose the dataset to validate (if several)

Execute the job


Defining jobs create a job2

Defining jobs: Create a job

When the validation isfinished the date isdisplayed

During the validation process, onlycancellationis possible

Validation is RUNNING


Defining jobs create a job3

Defining jobs: Create a job

Delete the job

Copy the job

When the validation isfinished the date isdisplayed

Click to view the results

Validation is COMPLETED


Defining jobs create a job4

VIEW RESULTS OF A JOB

Defining jobs: Create a job

Click to view the Error table


Defining jobs create a job5

VIEW ERROR TABLE OF A JOB

Filtering by Errorfields

Defining jobs: Create a job

Unfold Basic filtering

Unfold Advanced filtering

Error message number

Export the error table (CSV)


Defining jobs create a job6

Message containedinto the program

Severityusedinto the program

Name of the ruleinto the program

Defining jobs: Create a job

Rownumberwhere the erroroccuredinto the dataset

Click to viewdetails of the error

Variable values definedinto the program


Defining jobs create a job7

DETAILED VIEW OF ERROR

Defining jobs: Create a job

Select the datasetfileds to display

Error information

Dataset record (fieldsselected)


Defining jobs create a job8

EXPORT ERROR REPORT OF A JOB

Defining jobs: Create a job

Click to Export the error table in CSV format


Defining jobs create a job9

EXPORT ERROR REPORT OF A JOB

Choose CSV or FLR format

Defining jobs: Create a job

CSV parameters

Errorfieldsselected

Optionally, select Ascending or Descendingorder for anyerrorfield

Export table


Defining jobs create a job10

VIEW PROGRAM DETAILS

Defining jobs: Create a job

Content of the program


Defining jobs job statistics

Defining jobs: job statistics

Menu option

Job statistics


Defining jobs search job

Defining jobs: search job

Enter the searchcriteria

Delete the job

Copy the job

The corresponding jobs are displayed (all jobs if no selectedcriteria)

Click to view the results


E future developments

E - Future developments

  • Internationalisation – to offer the translation of the menus in other languages;

  • Gesmes full integration (registry);

  • SDMX 2.1 formats.


Useful links

Useful links

  • To EDIT page:http://ec.europa.eu/eurostat/edit

  • To VIPv page: CIRCAbc -> Eurostat -> VIP Validation Project

  • Generic data validation and editing service: mailto: [email protected]

  • EDIT as web – client - https://webgate.ec.europa.eu/eurostat/edit

  • CIRCAbc for:

    • EHSIS:https://circabc.europa.eu/w/browse/0b5ab24d-68a0-419f-a6bd-e41eb84f33fb

    • BoP:https://circabc.europa.eu/w/browse/01940df9-91ec-407b-9ba4-0f5c47086e0c

    • BoP:https://circabc.europa.eu/w/browse/ef8b542b-35a8-401c-9dd4-37f61e49f34d


Questions

Questions?

Thank you for your attention!


  • Login