(Solved) Assignment 1 – Data Cleaning and Visualisation | Homework Solution

Assignment 1 – Data Cleaning and Visualisation

Scenario

You have been provided an export from DCE’s incident response team’s security information and event management (SIEM) system. The incident response team extracted alert data from their SIEM platform and have provided a .CSV file (MLData2023.csv), with 500,000 event records, of which approximately 3,000 have been ‘tagged’ as malicious.

The goal is to integrate machine learning into their Security Information and Event Management (SIEM) platform so that suspicious events can be investigated in real-time. security data.

Data description

Each event record is a snapshot triggered by an individual network ‘packet’. The exact triggering conditions for the snapshot are unknown. But it is known that multiple packets are exchanged in a ‘TCP conversation’ between the source and the target before an event is triggered and a record created. It is also known that each event record is anomalous in some way Data Cleaning and Visualisation (the SIEM logs many events that may be suspicious).

A very small proportion of the data are known to be corrupted by their source systems and some data are incomplete or incorrectly tagged. The incident response team indicated this is likely to be less than a few hundred records. A list of the relevant features in the data is given below.

The raw data for the above variables are contained in the MLData2023.csv file.

Objectives

The data were gathered over a period of time and processed by several systems in order to associate specific events with confirmed malicious activities. However, the number of confirmed malicious events was very low, with these events accounting for less than 1% of all logged network events. Data Cleaning and Visualisation Get Myocardial Infarction Paper Help!!

Because the events associated with malicious traffic are quite rare, rate of ‘false negatives’ and ‘false positives’ are important.

Your initial goals will be to

• Perform some basic exploratory data analysis
• Clean the file and prepare it for Machine Learning (ML)
• Perform an initial Principal Component Analysis (PCA) of the
• Identify features that may be useful for ML algorithms
• Create a brief report to the rest of the research team on your findings

Task

First, copy the code below to a R script. Enter your student ID into the command set.seed(.)

and run the whole code. The code will create a sub-sample that is unique to you.

Use the str(.) command to check that the data type for each feature is correctly specified. Address the issue if this is not the case.

You are to clean and perform basic data analysis on the relevant features in mydata, and as well as principal component analysis (PCA) on the continuous variables. This is to be done using “R”. You will report on your findings.

Part 1 – Exploratory Data Analysis and Data Cleaning

• For each of your categorical or binary variables, determine the number (%) of instances for each of their categories and summarise them in a table as follows. Data Cleaning and Visualisation State all percentages in 1 decimal

• Summarise each of your continuous/numeric variables in a table as follows. State all values, except N, to 2 decimal places.

Note: The tables for subparts (i) and (ii) should be based on the original sub- sample of 600 observations, not the cleaned version.

• Examine the results in sub-parts (i) and (ii). Are there any invalid categories/values for the categorical variables? Is there any evidence of outliers for any of the continuous/numeric variables? Data Cleaning and Visualisation If so, how many and what percentage are there?

Part 2 – Perform PCA and Visualise Data

• For all the observations that you have deemed to be invalid/outliers in Part 1 (iii), mask them by replacing them with NAs using the replace(.) command in R. Data Cleaning and Visualisation
• Export your “cleaned” data as follows. This file will need to be submitted along with you #Write to a csv file.

write.csv(mydata,”mydata.csv”)

** Do not read the data back in and use them for PCA **

• Extract only the data for the numeric features in mydata, along with Class, and store them as a separate data frame/tibble. Then, filter the incomplete cases (i.e. any rows with NAs) and perform PCA using prcomp(.) in R, but only on the numeric features (i.e. exclude Class).
  • Outline why you believe the data should or should not be scaled, e. standardised, when performing PCA.
  • Outline the individual and cumulative proportions of variance (3 decimal places) explained by each of the first 4
  • Outline how many principal components (PCs) are adequate to explain at least 50% of the variability in your Data Cleaning and Visualisation
  • Outline the coefficients (or loadings) to 3 decimal places for PC1, PC2 and PC3, and describe which features (based on the loadings) are the key drivers for each of these three PCs.
• Create a biplot for PC1 vs PC2 to help visualise the results of your PCA in the first two Colour code the points with the variable Class. Write a paragraph to explain what your biplots are showing. That is, comment on the PCA plot, the loading plot individually, and then both plots combined (see Slides 28-29 of Module 3 notes) and outline and justify which (if any) of the features can help to distinguish Malicious events.
• Based on the results from parts (iii) to (iv), describe which dimension (have to choose one) can assist with the identification of Malicious events (Hint: project all the points in the PCA plot to PC1 axis and see whether there is good separation between the points for Malicious and Non-Malicious events. Then project to PC2 axis and see if there is separation between Malicious and Non-Malicious events, and whether it is better than the projection to.

What to Submit

1. A single report (not exceeding 5 pages, does not include cover page, contents page and reference page, if there is any) containing: Data Cleaning and Visualisation
   1. summary tables of all the variables in the dataset;
   2. a list of data issues (if any);
   3. your implementation of PCA and interpretation of the results, e. variances explained, and the contribution of each feature for PC1, PC2 and PC3;
   4. PC1 vs PC2 biplot and its interpretation;
   5. your explanation of selection and contribution of the features with respect to possible identification of Malicious.

If you use any references in your analysis or discussion outside of the notes provided in the unit, you must cite your sources.

1. The dataset containing your sub-sample of 600 observations, e., mydata.
2. A copy of your R

The report must be submitted through TURNITIN and checked for originality. The R code and data file are to be submitted separately via a Canvas submission link. Data Cleaning and Visualisation

Note that no marks will be given if the results you have provided cannot be confirmed by your code.

Marking Criteria

 Academic Misconduct

Edith Cowan University regards academic misconduct of any form as unacceptable. Academic misconduct, which includes but is not limited to, plagiarism; unauthorised collaboration; Data Cleaning and Visualisation cheating in examinations; theft of other student’s work; collusion; inadequate and incorrect referencing; will be dealt with in accordance with the ECU Rule for Academic Misconduct (including Plagiarism) Policy. Ensure that you are familiar with the Academic Misconduct Rules.

Assignment Extensions

Applications for extensions must be completed using the ECU Application for Extension form, which can be accessed online.

Normal work commitments, family commitments and extra-curricular activities are not accepted as grounds for granting you an extension as you are expected to plan ahead for your assessment due dates.

Please submit applications for extensions via email to both your tutor and the Unit Coordinator.

Where the assignment is submitted no more than 7 days late, the penalty shall, for each day that it is late, be 5% of the maximum assessment available for the assignment. Where the assignment is more than 7 days late, a mark of zero shall be awarded.