project | Joe Liang

ProtectYourVoice: Detacting AI Generated Voice using Deep Learning

Sat, 20 Apr 2024 00:00:00 +0000

Table of Contents

Disclaimer: This project, developed by Jiazhou Liang, Joe Liu, Tunghoi Yeung, and Yuechen Shi, is provided for non-commercial use. You are welcome to utilize the code for non-commercial purposes. Please note that the datasets incorporated in this project retain their original copyright held by their respective authors, as cited below.

Introduction

AI-generated (conversion) voices/speech have become increasingly popular, but they also present challenges in fraud prevention, such as impersonating someone to make fraudulent calls.

The problem that our deep fake voice detector aims to solve is the proliferation of audio-based misinformation and fraudulent activities facilitated by the advancement of deep learning techniques. With the rise of deep fake technology, individuals can manipulate audio recordings to create convincing fake voices that can be used for various malicious purposes, such as spreading false information, impersonating others, or committing fraud.

We think this project is crucial for safeguarding the authenticity and reliability of audio content, thereby preserving trust in communication channels and preventing harm caused by misinformation or malicious activities.

Methodology

To address this issue, we wish to implement a deep learning model to detect whether a piece of audio comes from a real person or is AI-generated, specifically focusing on the fake voices generated by deep learning models, e.g. Audio Deepfake (AD).

Current appoarch

Current approaches, such as the work from Bird et al., rely on classical statistical learning models like XGBoost and SVM for this task. Also, most approaches have various limitations, such as requiring special data processing to perform well, not resisting noises, and being limited to English speeches. Also, Bird et al. approach the dataset size as related to small. We aim to explore the performance of deep learning models in this problem domain, with a focus on resolving some limitations in the current approaches.

Pipline

The proposed approach involves four steps:

Converting the length of audio into a fixed length (30 seconds in this project).
Converting the raw audio (signal) into a spectrogram, which is a visual representation of its Fourier transformation.
Applying a CNN model, treating the spectrogram as an RGB image, to find the unique features within it.
Performing binary classification on these features to detect whether the audio is AI-generated or from a real human.

We will demonstrate the training process of this proposed approach, including dataset extraction, model selection, and performance evluation.

Import packages

# install missing packages in colab
# for audio segment
!pip install pydub
# for dataset download
!pip install -q kaggle

#import required packages
from pydub import AudioSegment
from scipy.io import wavfile
import soundfile as sf

from tqdm import tqdm
import os
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.model_selection import train_test_split
import matplotlib.image as mpimg

import os
import shutil
location = ''

# for better visualization
import warnings
warnings.filterwarnings("ignore")

# mount google drive folder
from google.colab import drive
drive.mount('/content/drive')
location = '/content/drive/MyDrive/shared-project-folder/'

Dataset

It is worth noting that there are various AD datasets available.

This project will use the “DEEP-VOICE: DeepFake Voice Recognition” dataset from Bird et al., as mentioned in the proposal, which was published on Kaggle. It is a smaller dataset that contains 64 raw audio files converted from the speeches of 8 public figures into other people’s voices using Retrieval-based Voice Conversion.

(Here is the illustration of how fake data is generated, according to Bird & Lotfi, 2023.) The average length of each sample is 600 seconds. To increase the sample size and reduce the length of each sample, we cut each sample into groups of 30-second clips.

While the Kaggle data contains audios coming from or mimicking a small number of celebrities, we would like our model to perform well on a wide variety of voices. ASVspoof 2019 was a database used for the Third Automatic Speaker Verification Spoofing and Countermeasures Challenge. We looked at a partition of ASVspoof 2019(within the “LA” folder in the original dataset) where the fake audios were generated by text-to-speech or voice conversion systems, from speech data captured from 107 speakers (46 males, 61 females) reading a list of text corpora. Here is the link to the original dataset: (https://datashare.ed.ac.uk/handle/10283/3336)

However, in contrast to the ‘Deep-Voice’ dataset, most of the audios are around 2 to 6 seconds long. We concatenate all the audios that have the same label (real or fake) and correspond to the same speaker together. In this way, the concatenated audios will have a closer length to the ones from Kaggle, and then we can split them into 30-second intervals in the same way.

Let us discuss the preprocessing of each dataset separately.

“DEEP-VOICE: DeepFake Voice Recognition” from kaggle

Download and process dataset

We need to download both dataset at the very first time, processed it, stored it in google drive for future usage.

from google.colab import files

#upload your kaggle api json (https://www.kaggle.com/docs/api#interacting-with-datasets)
files.upload()

!ls -lha kaggle.json
!mkdir -p ~/.kaggle
!cp kaggle.json ~/.kaggle/
!chmod 600 /root/.kaggle/kaggle.json
!pwd
!kaggle datasets download -d birdy654/deep-voice-deepfake-voice-recognition

unzip and processing dataset

!unzip /content/deep-voice-deepfake-voice-recognition.zip

fake_files = os.listdir('/content/KAGGLE/AUDIO/FAKE')
real_files = os.listdir('/content/KAGGLE/AUDIO/REAL')
print(f'Real voice samples: {len(real_files)}')
print(f'Fake voice samples: {len(fake_files)}')

Real voice samples: 8
Fake voice samples: 56

It worth noting that this is an imbalanced dataset (much more fake sample than real samples), we need to take this issue into account when eveluating the model.

audio = AudioSegment.from_wav('/content/KAGGLE/AUDIO/REAL/' + real_files[4])
length_audio = len(audio)
print(f'a random sample from real voice has {length_audio} millisecond')

a random sample from real voice has 600176 millisecond

We need to first convert the audio from wave into spetrogram before feed it into deep learning model. For ‘Deep-Voice’ dataset, as we metioned above, since each voice samples is relatively long (600 second) and overall sample size is small (64 samples), we want to utilizing data augmentation techniques to cut each sample into smaller segement to reduces the pixels per samples and increasing the sample size.

## some helper functions
def remove_all_files(folder_path):
 """
 Remove all old files in a folder and its subfolders.
 """
 for filename in os.listdir(folder_path):
 file_path = os.path.join(folder_path, filename)
 if os.path.isfile(file_path):
 os.remove(file_path)
 else:
 remove_all_files(file_path)

def split_audio(file_path, segment_length=30000, random_length = None) -> list:
 """
 Split the audio file into segments of given length (in milliseconds).
 Returns a list of audio segments.
 """
 audio = AudioSegment.from_wav(file_path)
 length_audio = len(audio)
 segments = []

 # Random Length: None or [lower, upper]
 # Generating a random segement length from given lower to upper range
 if random_length is not None:
 current_length = 0
 while current_length < len(audio):
 segment_length = np.random.randint(random_length[0],random_length[1],1)
 if current_length + segment_length >= len(audio):
 segment = audio[current_length:]
 else:
 segment = audio[current_length:current_length + segment_length]

 current_length += segment_length
 segments.append(segment)

 # fix length: cut samples based on given fixed segment_length
 else:
 for i in range(0, length_audio, segment_length):
 segment = audio[i:i + segment_length]
 segments.append(segment)

 return segments

Our audio-cutting function allows both fixed-length cutting (30 seconds by default) and random-length cutting (within a specified range) to accommodate real-world examples that different voice tends to have different length (leaving design space for possible future expansion). For now, we will cut each voice sample into fixed 30-second intervals and convert each interval into a spectrogram.

Converting real voice

# obtain all files from fake and real folders
fake_files = os.listdir('/content/KAGGLE/AUDIO/FAKE')
real_files = os.listdir('/content/KAGGLE/AUDIO/REAL')

os.makedirs('real',exist_ok=True)
#remove old files if exists
remove_all_files('real')
#iterating through all audio in the real files
for wav_file in real_files:
 print('Processing: ', wav_file)
 output_folder = 'real'
 #split to 30 seconds segments
 segments = split_audio('/content/KAGGLE/AUDIO/REAL/'+wav_file)
 for i, segment in enumerate(segments):
 segment = segment.set_channels(1)
 samples = np.array(segment.get_array_of_samples())
 #converting each segments into spectrograms
 specgram, freqs, times, im = plt.specgram(samples, Fs=segment.frame_rate, noverlap=512)
 plt.axis('off')
 #save the spectrograms for future usage
 plt.savefig(os.path.join('real', wav_file[:-5]+'_'+str(i+1)+'.jpg'), format="jpg")
 plt.close("all")

Processing: linus-original.wav
Processing: biden-original.wav
Processing: ryan-original.wav
Processing: margot-original.wav
Processing: taylor-original.wav
Processing: musk-original.wav
Processing: obama-original.wav
Processing: trump-original.wav

Convert fake voice

Using the same process as the real samples, we can convert all AI generated samples into the 30 seconds spectrogram

os.makedirs('fake',exist_ok=True)
#remove old files if exists
#remove_all_files('fake')
for wav_file in fake_files[46:]:
 print('Processing: ', wav_file)
 segments = split_audio('/content/KAGGLE/AUDIO/FAKE/'+wav_file)
 for i, segment in enumerate(segments):
 segment = segment.set_channels(1)
 samples = np.array(segment.get_array_of_samples())
 specgram, freqs, times, im = plt.specgram(samples, Fs=segment.frame_rate, NFFT=1024, noverlap=512)
 plt.axis('off')
 plt.savefig(os.path.join('fake', wav_file[:-5]+'_'+str(i+1)+'.jpg'), format="jpg")
 plt.close("all")

Processing: musk-to-taylor.wav
Processing: margot-to-taylor.wav
Processing: margot-to-ryan.wav
Processing: linus-to-musk.wav
Processing: ryan-to-trump.wav
Processing: taylor-to-biden.wav
Processing: trump-to-linus.wav
Processing: taylor-to-margot.wav
Processing: margot-to-obama.wav
Processing: biden-to-margot.wav

Some examples

Lets see some examples of the real and fake voices in the spectrograms

#coverted segment
real_spectro = os.listdir('/content/drive/MyDrive/shared-project-folder/kaggle/real')
fake_spectro = os.listdir('/content/drive/MyDrive/shared-project-folder/kaggle/fake')

# Display random five real and fake voice spectrogram
count = 0
fig, axs = plt.subplots(2,5)
for i in np.random.choice(range(len(real_spectro)),5):
 image_path = real_spectro[i]
 image = mpimg.imread(os.path.join('/content/drive/MyDrive/shared-project-folder/kaggle/real',image_path))
 axs[0,count].imshow(image)
 image_path = fake_spectro[i]
 image = mpimg.imread(os.path.join('/content/drive/MyDrive/shared-project-folder/kaggle/fake',image_path))
 axs[1,count].imshow(image)
 axs[0,count].axis('off')
 axs[1,count].axis('off')
 if count == 0:
 axs[0,count].set_title('real voice')
 axs[1,count].set_title('fake voice')
 count += 1

image.shape

(539, 542, 3)

The result spectrogram is represented as an RGB image with dimension 539 * 542. We will transformed it into a fix dimension for CNN model.

Train/Test splitting and regrouping samples into Pytorch ImageFolder structure

However, the spectrogram of real and fake voice segments are stored in two different folders. To utilize the convenient features in Torch’s ImageFolder, we have to split the dataset into train and test sets and convert them into the required format.

Moreover, we will spliting the dataset into train/test set in this steps, with 0.2 as the thersholds

from posix import remove
from sklearn.model_selection import train_test_split
from PIL import Image
# if the google drive is mounted, we can store the dataset into google drive for future usage
if os.path.exists('drive'):
 location = '/content/drive/MyDrive/shared-project-folder/kaggle'
else:
 # if not, stored it in runtime (required downloading processing data everytime)
 location = ''

if not os.path.exists(os.path.join(location,'train')):
 # Create the directory.
 os.makedirs(os.path.join(location,'train'))
 #real samples
 os.makedirs(os.path.join(location,'train','1'))
 #fake samples
 os.makedirs(os.path.join(location,'train','0'))

# combine all spectrogram
all_spectro = real_spectro + fake_spectro
# create label 1: real 0:fake
all_spectro_label = [1 for i in range(len(real_spectro))] + [0 for i in range(len(fake_spectro))]
#split
X_train, X_test, y_train, y_test = train_test_split(all_spectro, all_spectro_label, test_size=0.2, random_state=42)

for i in range(len(X_train)):
 y = y_train[i]
 destination_path = os.path.join(location,'train',str(y), X_train[i])
 with Image.open(['/content/drive/MyDrive/shared-project-folder/kaggle/fake/',
 '/content/drive/MyDrive/shared-project-folder/kaggle/real/'][y] + X_train[i]) as img:
 img.save(destination_path)

if not os.path.exists(os.path.join(location,'test')):
 os.makedirs(os.path.join(location,'test'))
 os.makedirs(os.path.join(location,'test','1'))
 os.makedirs(os.path.join(location,'test','0'))

for i in range(len(y_test)):
 y = y_test[i]
 destination_path = os.path.join(location,'test',str(y), X_test[i])
 with Image.open(['/content/drive/MyDrive/shared-project-folder/kaggle/fake/',
 '/content/drive/MyDrive/shared-project-folder/kaggle/real/'][y] + X_test[i]) as img:
 img.save(destination_path)

After cutting, converting into spectrogram, and re-ordering the file structure, we obtain a cleaned ‘Deep-Voice’ dataset that ready from model training. The cleaned dataset is stored in ‘kaggle’ folder of the shared google drive

ASVspoof 2019 from DataShare

Next, lets process the ASVspoof dataset, this dataset is much larger than the previous Kaggle dataset, so we need to download it directly into the colab and it will take sometimes

!wget https://datashare.ed.ac.uk/bitstream/handle/10283/3336/LA.zip

--2024-03-29 15:35:48-- https://datashare.ed.ac.uk/bitstream/handle/10283/3336/LA.zip
Resolving datashare.ed.ac.uk (datashare.ed.ac.uk)... 129.215.67.172
Connecting to datashare.ed.ac.uk (datashare.ed.ac.uk)|129.215.67.172|:443... connected.
HTTP request sent, awaiting response... 200 200
Length: 7640952520 (7.1G) [application/zip]
Saving to: ‘LA.zip’
LA.zip 100%[===================>] 7.12G 26.0MB/s in 5m 34s
2024-03-29 15:41:22 (21.8 MB/s) - ‘LA.zip’ saved [7640952520/7640952520]

!unzip LA.zip

Note: As the file paths in the following code cells indicated, we processed ASVspoof2019 LA locally. Processing on Google Colab would require large RAM and drive space, which have a high chance of crashing when using free colab account

Basic statistic of dataset

The ASVspoof dataset has a pre-defined train/validate/test split, we will not change its original split, so we retreives and process train/val/test data seperatly to ensure they are properly handle

training set

train_df = pd.read_csv(f'LA/ASVspoof2019_LA_cm_protocols/ASVspoof2019.LA.cm.train.trn.txt',
 sep=" ", header=None)
train_df.columns =['speaker_id','filename','system_id','null','class_name']
train_df.drop(columns=['null'],inplace=True)
train_df['filepath'] = f'LA/ASVspoof2019_LA_train/flac/'+train_df.filename+'.flac'

# fake audios are labeled 0, real (bona fide) are labeled 1
train_df['target'] = (train_df.class_name=='bonafide').astype('int32')
real_speakers = train_df['speaker_id'][train_df['target']==1].unique()
print(f'{len(real_speakers)} real speakers')
fake_speakers = train_df['speaker_id'][train_df['target']==0].unique()
print(f'{len(fake_speakers)} fake speakers')
print(f'{sum([f_speaker in real_speakers for f_speaker in fake_speakers])} common speakers')

20 real speakers
20 fake speakers
20 common speakers

There are 20 real speakers, and 20 fake speakers, and 20 of them are within both real and fake speaker, which mean all of speakers’ voice contains in both real and fake samples.

validation set

valid_df = pd.read_csv(f'LA/ASVspoof2019_LA_cm_protocols/ASVspoof2019.LA.cm.dev.trl.txt',
 sep=" ", header=None)
valid_df.columns =['speaker_id','filename','system_id','null','class_name']
valid_df.drop(columns=['null'],inplace=True)
valid_df['filepath'] = f'LA/ASVspoof2019_LA_dev/flac/'+valid_df.filename+'.flac'
# fake audios are labeled 0, real (bona fide) are labeled 1
valid_df['target'] = (valid_df.class_name=='bonafide').astype('int32')
real_speakers = valid_df['speaker_id'][valid_df['target']==1].unique()
print(f'{len(real_speakers)} real speakers')
fake_speakers = valid_df['speaker_id'][valid_df['target']==0].unique()
print(f'{len(fake_speakers)} fake speakers')
print(f'{sum([f_speaker in real_speakers for f_speaker in fake_speakers])} common speakers')

20 real speakers
10 fake speakers
10 common speakers

There are additional 10 speakers in the validation real dataset, which mean they do not have fake samples

test set

test_df = pd.read_csv(f'LA/ASVspoof2019_LA_cm_protocols/ASVspoof2019.LA.cm.eval.trl.txt',
 sep=" ", header=None)
test_df.columns =['speaker_id','filename','system_id','null','class_name']
test_df.drop(columns=['null'],inplace=True)
test_df['filepath'] = f'LA/ASVspoof2019_LA_eval/flac/'+test_df.filename+'.flac'
# fake audios are labeled 0, real (bona fide) are labeled 1
test_df['target'] = (test_df.class_name=='bonafide').astype('int32')
real_speakers = test_df['speaker_id'][test_df['target']==1].unique()
print(f'{len(real_speakers)} real speakers')
fake_speakers = test_df['speaker_id'][test_df['target']==0].unique()
print(f'{len(fake_speakers)} fake speakers')
print(f'{sum([f_speaker in real_speakers for f_speaker in fake_speakers])} common speakers')

67 real speakers
48 fake speakers
48 common speakers

Similiar for the validation dataset, there are 19 speakers does not have fake samples. Let us examine the length of some samples

for i in range(10):
 index = np.random.randint(0,train_df.shape[0], 1)
 path = train_df['filepath'][index]
 sound = AudioSegment.from_file(path.item(), format="flac")
 print("Lengths (in seconds) of samples: ",sound.duration_seconds)

Lengths (in seconds) of samples: 3.74325
Lengths (in seconds) of samples: 4.227125
Lengths (in seconds) of samples: 4.0983125
Lengths (in seconds) of samples: 2.2131875
Lengths (in seconds) of samples: 4.6369375
Lengths (in seconds) of samples: 3.8778125
Lengths (in seconds) of samples: 3.84675
Lengths (in seconds) of samples: 2.6333125
Lengths (in seconds) of samples: 1.569875
Lengths (in seconds) of samples: 1.510125

Combining samples

We observed that the length of each sample is quite short. To accommodate real-world scenarios, we decided to combine the voices from the same speaker together (fake with fake, real with real) to create longer voice segments.

from os.path import exists
# load the csv that contains the speaker id of each audio, and concat the audio into a single audio
def concat_audio(ref, dataset):
 speaker = ref['speaker_id'].unique()
 for current_speaker in speaker[27:]:
 print('Processing', current_speaker)
 real = np.empty((0,), dtype=np.int16)
 fake = np.empty((0,), dtype=np.int16)
 # group by speaker
 voice_from_speaker = ref[ref['speaker_id'] == current_speaker]
 for i, row in voice_from_speaker.iterrows():
 data,sample_rate = sf.read(row['filepath'])
 #group by label
 if row['target'] == 1:
 real = np.append(real, data)
 else:
 fake = np.append(fake, data)

 if real.shape[0] != 0:
 print('Real Exist')
 if dataset == 'train':
 out_path = 'LA_concated/train/real'
 elif dataset == 'validation':
 out_path = 'LA_concated/validation/real'
 else:
 out_path = 'LA_concated/test/real'

 os.makedirs(out_path, exist_ok=True)

 #combine and save
 sf.write(os.path.join(out_path,current_speaker+'.flac'), real, sample_rate)

 if fake.shape[0] != 0:
 print('Fake Exist')
 if dataset == 'train':
 out_path = 'LA_concated/train/fake'
 elif dataset == 'validation':
 out_path = 'LA_concated/validation/fake'
 else:
 out_path = 'LA_concated/test/fake'

 os.makedirs(out_path, exist_ok=True)
 sf.write(os.path.join(out_path,current_speaker+'.flac'), fake, sample_rate)

concatenate audio separately

concat_audio(train_df, 'train')

concat_audio(valid_df, 'validation')

concat_audio(test_df, 'test')

checking the length of concated voice segement in some samples:

files = os.listdir('LA_concated/train/real')
for i in range(5):
 index = np.random.randint(0,len(files), 1)
 path = files[index[0]]
 if path[-5:] == '.flac':
 sound = AudioSegment.from_file('LA_concated/train/real/' + path, format="flac")
 print("Lengths (in seconds) of real samples: ",sound.duration_seconds)
 sound = AudioSegment.from_file('LA_concated/train/fake/' + path, format="flac")
 print("Lengths (in seconds) of fake samples: ",sound.duration_seconds)

Lengths (in seconds) of real samples: 438.0238125
Lengths (in seconds) of fake samples: 4188.132875
Lengths (in seconds) of real samples: 424.0614375
Lengths (in seconds) of fake samples: 3624.6846875
Lengths (in seconds) of real samples: 350.6946875
Lengths (in seconds) of fake samples: 3809.648
Lengths (in seconds) of real samples: 458.1223125
Lengths (in seconds) of fake samples: 4087.7876875
Lengths (in seconds) of real samples: 439.1545625
Lengths (in seconds) of fake samples: 4109.019375

Converting into spectrogram

We have managed to concatenate the audios. Then we we can follow the same process as previous kaggle dataset to convert them into 30 secs segement and spectrogram

# this function similiar to previous splitting function, but using .flac files as input instead
def split_audio_flac(file_path, segment_length=30000, random_length = None):
 """
 Split the audio file into segments of given length (in milliseconds).
 Returns a list of audio segments.
 """
 audio = AudioSegment.from_file(file_path, format="flac")
 length_audio = len(audio)
 segments = []

 # Random Length: None or [lower, upper]
 # Generating a random segement length from lower to upper range
 if random_length is not None:
 current_length = 0
 while current_length < len(audio):
 segment_length = np.random.randint(random_length[0],random_length[1],1)
 if current_length + segment_length >= len(audio):
 segment = audio[current_length:]
 else:
 segment = audio[current_length:current_length + segment_length]

 current_length += segment_length
 segments.append(segment)

 # fix length: segment_length
 else:
 for i in range(0, length_audio, segment_length):
 segment = audio[i:i + segment_length]
 segments.append(segment)

 return segments

Training Set

we will first process training dataset in original ASVspoof 2019 contest

fake_files = os.listdir('LA_concated/train/fake')
real_files = os.listdir('LA_concated/train/real')

covert real voice into 30 second segement and spectrogram

os.makedirs('real',exist_ok=True)
#remove old files if exists
remove_all_files('real')
for wav_file in real_files:
 print('Processing: ', wav_file)
 output_folder = 'real'
 if wav_file[-5:] == '.flac':
 segments = split_audio_flac('LA_concated/train/real/'+wav_file)
 for i, segment in enumerate(segments):
 samples = np.array(segment.get_array_of_samples())
 specgram, freqs, times, im = plt.specgram(samples, Fs=segment.frame_rate, NFFT=1024, noverlap=512)
 plt.axis('off')
 plt.savefig(os.path.join('real', wav_file[:-5]+'_'+str(i+1)+'.jpg'), format="jpg")
 plt.close("all")

fake voice

os.makedirs('fake',exist_ok=True)
# remove_all_files('fake')
for wav_file in fake_files[12:]:
 print('Processing: ', wav_file)
 if wav_file[-5:] == '.flac':
 segments = split_audio_flac('LA_concated/train/fake/'+wav_file)
 for i, segment in enumerate(segments):
 samples = np.array(segment.get_array_of_samples())
 specgram, freqs, times, im = plt.specgram(samples, Fs=segment.frame_rate, NFFT=1024, noverlap=512)
 plt.axis('off')
 plt.savefig(os.path.join('fake', wav_file[:-5]+'_'+str(i+1)+'.jpg'), format="jpg")
 plt.close("all")

real_path = '/content/drive/MyDrive/shared-project-folder/LA_Spectrogram/train/1'
fake_path = '/content/drive/MyDrive/shared-project-folder/LA_Spectrogram/train/0'
real_spectro = os.listdir(real_path)
fake_spectro = os.listdir(fake_path)
print(f'There are {len(real_spectro)} real samples in training set, each contain 30 seconds of voices in spectrogram')
print(f'There are {len(fake_spectro)} fake samples in training set')

There are 303 real samples in training set, each contain 30 seconds of voices in spectrogram
There are 2615 fake samples in training set

Note: this is still an imbalanced dataset

some samples

# Display random five real and fake voice spectrogram
count = 0
fig, axs = plt.subplots(2,5)
for i in np.random.choice(range(len(real_spectro)),5):
 image_path = real_spectro[i]
 image = mpimg.imread(os.path.join(real_path,image_path))
 axs[0,count].imshow(image)
 image_path = fake_spectro[i]
 image = mpimg.imread(os.path.join(fake_path,image_path))
 axs[1,count].imshow(image)
 axs[0,count].axis('off')
 axs[1,count].axis('off')
 if count == 0:
 axs[0,count].set_title('real voice')
 axs[1,count].set_title('fake voice')
 count += 1

Test and Validation Set

Similiarly, we also converted the test set into the spectrogram using the same process as train set

fake_files = os.listdir('LA_concated/test/fake')
real_files = os.listdir('LA_concated/test/real')
#converting real
output_folder = 'LA_Spectrogram/test/1'
os.makedirs(output_folder,exist_ok=True)
for wav_file in real_files:
 print('Processing: ', wav_file)
 if wav_file[-5:] == '.flac':
 segments = split_audio_flac('LA_concated/test/real/'+wav_file)
 for i, segment in enumerate(segments):
 samples = np.array(segment.get_array_of_samples())
 specgram, freqs, times, im = plt.specgram(samples, Fs=segment.frame_rate, NFFT=1024, noverlap=512)
 plt.axis('off')
 plt.savefig(os.path.join(output_folder, wav_file[:-5]+'_'+str(i+1)+'.jpg'), format="jpg")
 plt.close("all")

#converting fake
output_folder = 'LA_Spectrogram/test/0'
os.makedirs(output_folder,exist_ok=True)
for wav_file in fake_files[11:]:
 print('Processing: ', wav_file)
 if wav_file[-5:] == '.flac':
 segments = split_audio_flac('LA_concated/test/fake/'+wav_file)
 for i, segment in enumerate(segments):
 samples = np.array(segment.get_array_of_samples())
 specgram, freqs, times, im = plt.specgram(samples, Fs=segment.frame_rate, NFFT=1024, noverlap=512)
 plt.axis('off')
 plt.savefig(os.path.join(output_folder, wav_file[:-5]+'_'+str(i+1)+'.jpg'), format="jpg")
 plt.close("all")

validation set as well

fake_files = os.listdir(location + 'LA_concated/validation/fake')
real_files = os.listdir(location + 'LA_concated/validation/real')
#converting real
output_folder = 'LA_Spectrogram/validation/1'
os.makedirs(output_folder,exist_ok=True)
for wav_file in real_files:
 print('Processing: ', wav_file)
 if wav_file[-5:] == '.flac':
 segments = split_audio_flac(location + 'LA_concated/validation/real/'+wav_file)
 for i, segment in enumerate(segments):
 samples = np.array(segment.get_array_of_samples())
 specgram, freqs, times, im = plt.specgram(samples, Fs=segment.frame_rate, NFFT=1024, noverlap=512)
 plt.axis('off')
 plt.savefig(os.path.join(output_folder, wav_file[:-5]+'_'+str(i+1)+'.jpg'), format="jpg")
 plt.close("all")

#converting fake
output_folder = 'LA_Spectrogram/validation/0'
os.makedirs(output_folder,exist_ok=True)
for wav_file in fake_files[7:]:
 print('Processing: ', wav_file)
 if wav_file[-5:] == '.flac':
 segments = split_audio_flac(location + 'LA_concated/validation/fake/'+wav_file)
 for i, segment in enumerate(segments):
 samples = np.array(segment.get_array_of_samples())
 specgram, freqs, times, im = plt.specgram(samples, Fs=segment.frame_rate, NFFT=1024, noverlap=512)
 plt.axis('off')
 plt.savefig(os.path.join(output_folder, wav_file[:-5]+'_'+str(i+1)+'.jpg'), format="jpg")
 plt.close("all")

All samples are pre-splited which allows us to using this way to converting dataset, and the file structure is ready for torchvison to use

Unseen samples for model evaluation

We need to evaluate the model performance on unseen samples.

We would use another dataset which consists of original voice audios and fake audios obtained by imitation using Efficient Wavelet Masking (Rodríguez, 2019) to evaluate our final model.

This dataset was processed into spectrograms in the same way as previously. But we want to keep the length as its original length, which is not fixed 30 seconds, since this is more fit to the real world usage.

real_unseen = os.listdir('/content/drive/MyDrive/shared-project-folder/Fake_voice_recordings_Imitation/real')
fake_unseen = os.listdir('/content/drive/MyDrive/shared-project-folder/Fake_voice_recordings_Imitation/fake')
#remove old files
remove_all_files('/content/drive/MyDrive/shared-project-folder/unseen/test/real')
remove_all_files('/content/drive/MyDrive/shared-project-folder/unseen/test/fake')
for file_path in real_unseen:
 if file_path[-4:] == '.wav':
 segments = split_audio('/content/drive/MyDrive/shared-project-folder/Fake_voice_recordings_Imitation/real/'+file_path)
 # the audios have various lengths and are shorter than 30 seconds,
 # so split_audio just keeps the whole sequence without creating 30-second intervals
 for i, segment in enumerate(segments):
 segment = segment.set_channels(1)
 samples = np.array(segment.get_array_of_samples())
 specgram, freqs, times, im = plt.specgram(samples, Fs=segment.frame_rate)
 im.axes.get_xaxis().set_visible(False)
 im.axes.get_yaxis().set_visible(False)
 plt.axis('off')
 output_folder = os.path.join('/content/drive/MyDrive/shared-project-folder/unseen/', 'test', '1')
 os.makedirs(os.path.join(output_folder),exist_ok=True)
 plt.savefig(os.path.join(output_folder, file_path[:-4]+'_'+str(i+1)+'.jpg'), format="jpg")
 plt.close("all")

for file_path in fake_unseen:
 if file_path[-4:] == '.wav':
 segments = split_audio('/content/drive/MyDrive/shared-project-folder/Fake_voice_recordings_Imitation/fake/'+file_path)
 for i, segment in enumerate(segments):
 segment = segment.set_channels(1)
 samples = np.array(segment.get_array_of_samples())
 specgram, freqs, times, im = plt.specgram(samples, Fs=segment.frame_rate)
 im.axes.get_xaxis().set_visible(False)
 im.axes.get_yaxis().set_visible(False)
 #change y range
 plt.axis('off')
 output_folder = os.path.join('/content/drive/MyDrive/shared-project-folder/unseen/', 'test', '0')
 os.makedirs(os.path.join(output_folder),exist_ok=True)
 plt.savefig(os.path.join(output_folder, file_path[:-4]+'_'+str(i+1)+'.jpg'), format="jpg")
 plt.close("all")

Lets see some samples in unseen samples

real_path = '/content/drive/MyDrive/shared-project-folder/unseen/test/1'
fake_path = '/content/drive/MyDrive/shared-project-folder/unseen/test/0'
real_spectro = os.listdir(real_path)
fake_spectro = os.listdir(fake_path)

count = 0
fig, axs = plt.subplots(2,5)
for i in np.random.choice(range(len(real_spectro)),5):
 image_path = real_spectro[i]
 image = mpimg.imread(os.path.join(real_path,image_path))
 axs[0,count].imshow(image)
 image_path = fake_spectro[i]
 image = mpimg.imread(os.path.join(fake_path,image_path))
 axs[1,count].imshow(image)
 axs[0,count].axis('off')
 axs[1,count].axis('off')
 if count == 0:
 axs[0,count].set_title('real voice')
 axs[1,count].set_title('fake voice')
 count += 1

Model training and eveluation

We first defined some helper functions which created dataloaders to access training data and validation data, and computed the accuracy and the F1 score of the model.

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F

from torchvision import datasets, transforms
from torch.utils.data import DataLoader
from torch.utils.data import random_split

def spliting_data(path, batch_size, split = 0.2):
 # if the google drive is mounted, we can store the dataset
 # into google drive for future usage (make sure creating a shortcut of 'shared
 # project folder'in 'My Drive')

 if os.path.exists('drive'):
 location = '/content/drive/MyDrive/shared-project-folder/'
 else:
 # if not, stored it in runtime (required downloading processing data everytime)
 location = ''

 #kaggle dataset
 data_dir = location + path

 #resize the spectrogram
 transform = transforms.Compose([
 transforms.Resize((224, 224)),
 transforms.ToTensor(),
 ])
 print('Reading Data')
 #using both dataset
 if path == 'both':
 dataset1 = datasets.ImageFolder('/content/drive/MyDrive/shared-project-folder/LA_Spectrogram/train', transform=transform)
 val_dataset1 = datasets.ImageFolder('/content/drive/MyDrive/shared-project-folder/LA_Spectrogram/validation', transform=transform)

 #kaggle dataset does not pre-split
 dataset2 = datasets.ImageFolder('/content/drive/MyDrive/shared-project-folder/kaggle/train', transform=transform)
 train_size = int((1-split) * len(dataset2))
 val_size = len(dataset2) - train_size
 dataset2, val_dataset2 = random_split(dataset2, [train_size, val_size])

 train_dataset = torch.utils.data.ConcatDataset([dataset1, dataset2])
 val_dataset = torch.utils.data.ConcatDataset([val_dataset1, val_dataset2])

 # using single one dataset
 else:
 if path == 'kaggle':
 #using kaggle
 dataset = datasets.ImageFolder('/content/drive/MyDrive/shared-project-folder/kaggle/train', transform=transform)
 train_size = int((1-split) * len(dataset))
 al_size = len(dataset2) - train_size
 train_dataset, val_dataset = random_split(dataset, [train_size, val_size])
 else:
 #using asvsproof
 train_dataset = datasets.ImageFolder('/content/drive/MyDrive/shared-project-folder/LA_Spectrogram/train', transform=transform)
 val_dataset = datasets.ImageFolder('/content/drive/MyDrive/shared-project-folder/LA_Spectrogram/validation', transform=transform)

 print(f'Training Sample: {len(train_dataset)}')
 print(f'Validation Sample: {len(val_dataset)}')

 train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
 val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=True)

 return train_loader, val_loader

# evalation the performance
def evaluate(model, test_loader):
 model.eval()
 device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
 test_loss = 0
 test_accuracy = 0
 test_f1 = 0
 criterion = nn.CrossEntropyLoss()
 with torch.no_grad():
 for inputs, labels in test_loader:
 inputs, labels = inputs.to(device), labels.to(device)
 outputs = model(inputs)
 test_loss += criterion(outputs, labels)
 _, predicted = torch.max(outputs.data, 1)
 labels = labels.cpu()
 predicted = predicted.cpu()
 test_accuracy += accuracy_score(labels, predicted)
 test_f1 += f1_score(labels, predicted)

 return test_loss.detach().cpu() / len(test_loader), test_f1 / len(test_loader),test_accuracy / len(test_loader)

Also, another helper function for training

from sklearn.metrics import f1_score
from sklearn.metrics import accuracy_score

def train(model, dataset_path, epoches = 30, learning_rate = 0.0001,
 batch_size = 32, check_point = True, random_state = 42, plot = True):
 torch.manual_seed(random_state)
 model = model
 train_loader, val_loader = spliting_data(dataset_path,batch_size)

 # Assuming you have your train_loader set up as before
 criterion = nn.CrossEntropyLoss()
 optimizer = optim.Adam(model.parameters(), lr=learning_rate)

 # Move the model to the GPU if available
 device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

 model.to(device)

 num_epochs = epoches
 training_loss = []
 training_accuracy = []
 training_f1 = []
 validation_loss = []
 validation_accuracy = []
 validation_f1 = []

 for epoch in tqdm(range(num_epochs), leave = False, desc = f'Epoch'):
 # Set the model to training mode
 model.train()
 running_loss = 0.0
 correct = 0
 total = 0
 f1 = 0
 accuracy = 0

 for inputs, labels in train_loader:
 inputs, labels = inputs.to(device), labels.to(device)
 optimizer.zero_grad()
 outputs = model(inputs)
 loss = criterion(outputs, labels)
 loss.backward()
 optimizer.step()

 running_loss += loss.item()

 # Calculate accuracy
 _, predicted = torch.max(outputs.data, 1)
 labels = labels.cpu()
 predicted = predicted.cpu()
 accuracy += accuracy_score(labels, predicted)
 f1 += f1_score(labels, predicted)

 epoch_loss = running_loss / len(train_loader)
 epoch_f1 = f1 / len(train_loader)
 epoch_accuracy = accuracy/ len(train_loader)

 #print, eval, and store every 2:
 if epoch % 2 == 0:
 training_loss.append(epoch_loss)
 training_accuracy.append(epoch_accuracy)
 training_f1.append(epoch_f1)

 epoch_val_loss, epoch_val_f1, epoch_val_accuracy = evaluate(model, val_loader)

 validation_loss.append(epoch_val_loss)
 validation_accuracy.append(epoch_val_accuracy)
 validation_f1.append(epoch_val_f1)

 #save checkpoint
 if check_point:
 checkpoint = {'epoch': epoch,'model_state_dict': model.state_dict(),
 'optimizer_state_dict': optimizer.state_dict(),'loss': loss}
 out_path = dataset_path.split('/')[0]+f'/{model.name}/lr_{learning_rate}_batch_{batch_size}_new'
 os.makedirs(location + out_path,exist_ok=True)
 checkpoint_filename = f'checkpoint_epoch{epoch+1}.pth'
 torch.save(checkpoint, os.path.join(location + out_path,checkpoint_filename))

 print(f'''
 Epoch {epoch+1}/{num_epochs}
 Train Loss: {epoch_loss:.4f}, Train Accuracy: {epoch_accuracy:.2f}, Train F1: {epoch_f1:.2f}
 Val Loss: {epoch_val_loss:.4f}, Val Accuracy: {epoch_val_accuracy:.2f}, Val F1: {epoch_val_f1:.2f}
 ''')

 if plot:
 # Plot the training loss and accuracy curves
 plt.figure(figsize=(18, 6))
 plt.subplot(1, 3, 1)
 plt.plot(range(1, num_epochs + 1,2), training_loss, label='Training Loss')
 plt.plot(range(1, num_epochs + 1,2), validation_loss, label='Validation Loss')
 plt.xlabel('Epochs')
 plt.ylabel('Loss')
 plt.title('Training Loss Curve')
 plt.legend()

 plt.subplot(1, 3, 2)
 plt.plot(range(1, num_epochs + 1,2), training_accuracy, label='Training Accuracy')
 plt.plot(range(1, num_epochs + 1,2), validation_accuracy, label='Validation Accuracy')
 plt.xlabel('Epochs')
 plt.ylabel('Accuracy')
 plt.title('Training Accuracy Curve')
 plt.legend()

 plt.subplot(1, 3, 3)
 plt.plot(range(1, num_epochs + 1,2), training_f1, label='Training Accuracy')
 plt.plot(range(1, num_epochs + 1,2), validation_f1, label='Validation Accuracy')
 plt.xlabel('Epochs')
 plt.ylabel('F1 Score')
 plt.title('F1 Score Curve')
 plt.legend()
 plt.show()

 return model

Baseline Model

We first trained a CNN without transferred learning from ResNet as the baseline model. This CNN simply had 2 convolution layers each followed by max-pooling, and 3 fully-connected layers. Activation function between the 3 fully-connected layers was ReLu.

class SimpleCNN(nn.Module):
 def __init__(self):
 super(SimpleCNN, self).__init__()
 self.name = "SimpleCNN"
 self.conv1 = nn.Conv2d(3, 6, 5)
 self.pool = nn.MaxPool2d(2, 2)
 self.conv2 = nn.Conv2d(6, 16, 5)
 self.fc1 = nn.Linear(16 * 53 * 53, 120)
 self.fc2 = nn.Linear(120, 84)
 self.fc3 = nn.Linear(84, 2)

 def forward(self, x):
 x = self.pool(F.relu(self.conv1(x)))
 x = self.pool(F.relu(self.conv2(x)))
 x = x.view(-1, 16 * 53 * 53)
 x = F.relu(self.fc1(x))
 x = F.relu(self.fc2(x))
 x = self.fc3(x)
 return x

Training

model = SimpleCNN()
# 'both' will combine two dataset,
#' kaggle/train' will only used kaggle dataset
# 'LA_Spectrogram/train' will used ASVsproof dataset

train(model, 'both',batch_size=64,learning_rate=0.001)

Reading Data
Training Sample: 2986
Validation Sample: 747
Epoch: 3%|▎ | 1/30 [05:24<2:36:47, 324.39s/it]
Epoch 1/30
Train Loss: 0.3613, Train Accuracy: 0.89, Train F1: 0.00
Val Loss: 0.3500, Val Accuracy: 0.89, Val F1: 0.00
Epoch: 10%|█ | 3/30 [06:35<45:24, 100.90s/it]
Epoch 3/30
Train Loss: 0.3125, Train Accuracy: 0.90, Train F1: 0.06
Val Loss: 0.3126, Val Accuracy: 0.91, Val F1: 0.27
Epoch: 17%|█▋ | 5/30 [07:48<25:43, 61.74s/it]
Epoch 5/30
Train Loss: 0.2594, Train Accuracy: 0.91, Train F1: 0.26
Val Loss: 0.2729, Val Accuracy: 0.90, Val F1: 0.27
Epoch: 23%|██▎ | 7/30 [08:56<17:52, 46.63s/it]
Epoch 7/30
Train Loss: 0.1809, Train Accuracy: 0.93, Train F1: 0.55
Val Loss: 0.2037, Val Accuracy: 0.92, Val F1: 0.45
Epoch: 30%|███ | 9/30 [10:04<14:10, 40.49s/it]
Epoch 9/30
Train Loss: 0.0904, Train Accuracy: 0.97, Train F1: 0.80
Val Loss: 0.1039, Val Accuracy: 0.96, Val F1: 0.81
Epoch: 37%|███▋ | 11/30 [11:14<12:00, 37.92s/it]
Epoch 11/30
Train Loss: 0.0697, Train Accuracy: 0.97, Train F1: 0.86
Val Loss: 0.0878, Val Accuracy: 0.96, Val F1: 0.82
Epoch: 43%|████▎ | 13/30 [12:23<10:18, 36.41s/it]
Epoch 13/30
Train Loss: 0.0398, Train Accuracy: 0.99, Train F1: 0.92
Val Loss: 0.0806, Val Accuracy: 0.96, Val F1: 0.83
Epoch: 50%|█████ | 15/30 [13:32<08:58, 35.90s/it]
Epoch 15/30
Train Loss: 0.0238, Train Accuracy: 0.99, Train F1: 0.97
Val Loss: 0.0811, Val Accuracy: 0.97, Val F1: 0.83
Epoch: 57%|█████▋ | 17/30 [14:41<07:41, 35.52s/it]
Epoch 17/30
Train Loss: 0.0092, Train Accuracy: 1.00, Train F1: 0.99
Val Loss: 0.0564, Val Accuracy: 0.98, Val F1: 0.89
Epoch: 63%|██████▎ | 19/30 [15:49<06:26, 35.15s/it]
Epoch 19/30
Train Loss: 0.0018, Train Accuracy: 1.00, Train F1: 1.00
Val Loss: 0.0630, Val Accuracy: 0.98, Val F1: 0.89
Epoch: 70%|███████ | 21/30 [17:00<05:21, 35.68s/it]
Epoch 21/30
Train Loss: 0.0007, Train Accuracy: 1.00, Train F1: 1.00
Val Loss: 0.0677, Val Accuracy: 0.98, Val F1: 0.90
Epoch: 77%|███████▋ | 23/30 [18:09<04:06, 35.25s/it]
Epoch 23/30
Train Loss: 0.0004, Train Accuracy: 1.00, Train F1: 1.00
Val Loss: 0.0718, Val Accuracy: 0.98, Val F1: 0.91
Epoch: 83%|████████▎ | 25/30 [19:17<02:54, 34.99s/it]
Epoch 25/30
Train Loss: 0.0003, Train Accuracy: 1.00, Train F1: 1.00
Val Loss: 0.0697, Val Accuracy: 0.98, Val F1: 0.89
Epoch: 90%|█████████ | 27/30 [20:30<01:47, 35.86s/it]
Epoch 27/30
Train Loss: 0.0002, Train Accuracy: 1.00, Train F1: 1.00
Val Loss: 0.0737, Val Accuracy: 0.98, Val F1: 0.90
Epoch: 97%|█████████▋| 29/30 [21:41<00:35, 35.89s/it]
Epoch 29/30
Train Loss: 0.0002, Train Accuracy: 1.00, Train F1: 1.00
Val Loss: 0.0744, Val Accuracy: 0.98, Val F1: 0.89

SimpleCNN(
(conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1))
(pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))
(fc1): Linear(in_features=44944, out_features=120, bias=True)
(fc2): Linear(in_features=120, out_features=84, bias=True)
(fc3): Linear(in_features=84, out_features=2, bias=True)
)

The highest validation F1 we got was 0.91 and the corresponding accuracy was 0.98. A validation F1 of 0.91 is very good (Buhl, 2023), but by tuning hyperparameters based on validation F1, the model might have overfit on the validation data, so we should check the performance on the test data.

Evaluation

We loaded the trained baseline model.

model = SimpleCNN()
#load the checkpoint with lowest F1
path = location + '/both/SimpleCNN/lr_0.001_batch_64/checkpoint_epoch23.pth'
checkpoint = torch.load(path)
model_state_dict = checkpoint['model_state_dict']
model.load_state_dict(model_state_dict)

<All keys matched successfully>

Then we evaluated the model’s performance on the test data. Note that the test data consisted of test samples from both “Deep-Voice” on Kaggle and ASVspoof2019 LA.

transform = transforms.Compose([
 transforms.Resize((224, 224)),
 transforms.ToTensor(),
])

test_dataset1 = datasets.ImageFolder((location + 'kaggle/test'), transform=transform)
test_dataset2 = datasets.ImageFolder((location + 'LA_Spectrogram/test'), transform=transform)
test_dataset = torch.utils.data.ConcatDataset([test_dataset1, test_dataset2])
test_loader = DataLoader(test_dataset, batch_size=64, shuffle=True)

loss, f1, accuracy = evaluate(model, test_loader)
print(f'The F1-Score for Testing Dataset is {f1}')
print(f'The accuracy for Testing Dataset is {accuracy}')

The F1-Score for Testing Dataset is 0.7845597579699278
The accuracy for Testing Dataset is 0.9271012931034484

Although the test accuracy of baseline model was high (0.927), the F1 score for the testing dataset was relatively low (0.785). This means that the model didn’t achieve an as satisfying balance between precision and recall as it did on the validation set. Since our training,validation and test data all had more fake samples than real samples, the baseline model might be classifying samples as fake without learning sufficient features.

Transfer Learning: Pre-train Res-Net

Our next step was to build a model that utilized transferred learning from ResNet18. The architecture was composed of 1 convolution layer followed by the pretrained ResNet18, and 2 fully-connected layers in the end. We did not freeze the parameters from ResNet18. In other words, they were subject to update by backpropagation. Our justification was that ResNet18 was pretrained on ImageNet which did not contain any spectrograms or other signal visualizations.

import torch
import torchvision.models as models
import torch.nn as nn

class PreTrainedResNet18(nn.Module):
 def __init__(self):
 super().__init__()
 self.name = 'PreRes'
 self.conv1 = nn.Conv2d(3, 3, kernel_size=7, stride=2, padding=3, bias=False)
 self.resnet = models.resnet18(pretrained=True)
 self.fc1 = nn.Linear(1000, 100)
 self.fc2 = nn.Linear(100, 2)
 self.activation = nn.ReLU()
 self.dropout = nn.Dropout(p=0.1)

 # Copy the pre-trained model weights except for the first convolution layer
 state_dict = self.resnet.state_dict()
 for key in list(state_dict.keys()):
 if not key.startswith('conv1'):
 self.state_dict()[key] = state_dict[key]

 def forward(self, x):
 x = self.conv1(x)
 x = self.resnet(x)
 x = self.activation(self.fc1(x))
 x = self.dropout(x)
 x = self.fc2(x)
 return x

Model Training and Fine-tuning

# training
model_resnet = PreTrainedResNet18()
model_resnet = train(model_resnet, 'both',batch_size=256,learning_rate=0.0001)

Downloading: "https://download.pytorch.org/models/resnet18-f37072fd.pth" to /root/.cache/torch/hub/checkpoints/resnet18-f37072fd.pth
100%|██████████| 44.7M/44.7M [00:00<00:00, 120MB/s]
Reading Data
Training Sample: 2986
Validation Sample: 747
Epoch: 3%|▎ | 1/30 [01:09<33:42, 69.73s/it]
Epoch 1/30
Train Loss: 0.3109, Train Accuracy: 0.86, Train F1: 0.31
Val Loss: 0.3418, Val Accuracy: 0.89, Val F1: 0.00
Epoch: 10%|█ | 3/30 [02:22<20:03, 44.57s/it]
Epoch 3/30
Train Loss: 0.0240, Train Accuracy: 0.99, Train F1: 0.97
Val Loss: 0.2347, Val Accuracy: 0.91, Val F1: 0.35
Epoch: 17%|█▋ | 5/30 [03:34<16:26, 39.48s/it]
Epoch 5/30
Train Loss: 0.0028, Train Accuracy: 1.00, Train F1: 1.00
Val Loss: 0.0516, Val Accuracy: 0.98, Val F1: 0.91
Epoch: 23%|██▎ | 7/30 [04:44<14:23, 37.53s/it]
Epoch 7/30
Train Loss: 0.0004, Train Accuracy: 1.00, Train F1: 1.00
Val Loss: 0.0246, Val Accuracy: 0.99, Val F1: 0.96
Epoch: 30%|███ | 9/30 [05:57<13:00, 37.17s/it]
Epoch 9/30
Train Loss: 0.0001, Train Accuracy: 1.00, Train F1: 1.00
Val Loss: 0.0205, Val Accuracy: 1.00, Val F1: 0.98
Epoch: 37%|███▋ | 11/30 [07:08<11:36, 36.66s/it]
Epoch 11/30
Train Loss: 0.0001, Train Accuracy: 1.00, Train F1: 1.00
Val Loss: 0.0209, Val Accuracy: 0.99, Val F1: 0.97
Epoch: 43%|████▎ | 13/30 [08:20<10:23, 36.66s/it]
Epoch 13/30
Train Loss: 0.0001, Train Accuracy: 1.00, Train F1: 1.00
Val Loss: 0.0204, Val Accuracy: 0.99, Val F1: 0.96
Epoch: 50%|█████ | 15/30 [09:32<09:07, 36.53s/it]
Epoch 15/30
Train Loss: 0.0001, Train Accuracy: 1.00, Train F1: 1.00
Val Loss: 0.0209, Val Accuracy: 0.99, Val F1: 0.97
Epoch: 57%|█████▋ | 17/30 [10:43<07:52, 36.37s/it]
Epoch 17/30
Train Loss: 0.0001, Train Accuracy: 1.00, Train F1: 1.00
Val Loss: 0.0198, Val Accuracy: 0.99, Val F1: 0.97
Epoch: 63%|██████▎ | 19/30 [11:59<06:58, 38.02s/it]
Epoch 19/30
Train Loss: 0.0001, Train Accuracy: 1.00, Train F1: 1.00
Val Loss: 0.0207, Val Accuracy: 0.99, Val F1: 0.97
Epoch: 70%|███████ | 21/30 [13:10<05:33, 37.06s/it]
Epoch 21/30
Train Loss: 0.0000, Train Accuracy: 1.00, Train F1: 1.00
Val Loss: 0.0204, Val Accuracy: 0.99, Val F1: 0.97
Epoch: 77%|███████▋ | 23/30 [14:21<04:16, 36.60s/it]
Epoch 23/30
Train Loss: 0.0000, Train Accuracy: 1.00, Train F1: 1.00
Val Loss: 0.0210, Val Accuracy: 0.99, Val F1: 0.96
Epoch: 83%|████████▎ | 25/30 [15:33<03:02, 36.54s/it]
Epoch 25/30
Train Loss: 0.0000, Train Accuracy: 1.00, Train F1: 1.00
Val Loss: 0.0209, Val Accuracy: 0.99, Val F1: 0.97
Epoch: 90%|█████████ | 27/30 [16:46<01:49, 36.60s/it]
Epoch 27/30
Train Loss: 0.0000, Train Accuracy: 1.00, Train F1: 1.00
Val Loss: 0.0216, Val Accuracy: 0.99, Val F1: 0.97
Epoch: 97%|█████████▋| 29/30 [17:56<00:36, 36.26s/it]
Epoch 29/30
Train Loss: 0.0000, Train Accuracy: 1.00, Train F1: 1.00
Val Loss: 0.0222, Val Accuracy: 0.99, Val F1: 0.97

The highest validation F1 was 0.98, and the validation accuracy was approximately 1. This was an improvement of the baseline model. Comparing the training curves, we also saw that in terms of epochs, the ResNet18 model was converging faster and smoother. Then we checked its performance on the test data.

# load the best model
model_resnet = PreTrainedResNet18()
path = location+'both/PreRes/lr_0.0001_batch_256/checkpoint_epoch9.pth'
checkpoint = torch.load(path, map_location=torch.device('cpu'))
model_state_dict = checkpoint['model_state_dict']
model_resnet.load_state_dict(model_state_dict)

Downloading: "https://download.pytorch.org/models/resnet18-f37072fd.pth" to /root/.cache/torch/hub/checkpoints/resnet18-f37072fd.pth
100%|██████████| 44.7M/44.7M [00:00<00:00, 77.1MB/s]
<All keys matched successfully>

# evaluate by the test dataset
loss, f1, accuracy = evaluate(model_resnet, test_loader)
print(f'The F1-Score for Testing Dataset is {f1}')
print(f'The accuracy for Testing Dataset is {accuracy}')

The F1-Score for Testing Dataset is 0.8942586360721817
The accuracy for Testing Dataset is 0.9690193965517241

While the test accuracies were close (0.969 v.s. 0.927), test F1 of ResNet18 model was quite higher (0.894 v.s. 0.785), so utilizing ResNet18 helped us achieve better balance between precision and recall. Our final model would be the ResNet18 model.

Unseen samples

At the end of model training, we wish to test our best model with the unseen samples we prepared before. There are 50 fake samples, and 50 real samples.

test_dataset = datasets.ImageFolder(('/content/drive/MyDrive/shared-project-folder/unseen/test'), transform=transform)
test_loader = DataLoader(test_dataset, batch_size=64, shuffle=True)
loss, f1, accuracy = evaluate(model_resnet, test_loader)
print(f'The F1-Score for Testing Dataset is {f1}')
print(f'The accuracy for Testing Dataset is {accuracy}')

The F1-Score for Testing Dataset is 0.8282732447817838
The accuracy for Testing Dataset is 0.8402777777777778

At this point we tested our final ResNet18 model on new spectrograms. It was expectable that the F1 score and the accuracy decrease as the audios from which the new spectrograms were generated in one unique method, which was likely not included in ASVspoof2019 LA (the generation methods were labeled anonymously). The F1 score didn’t drop too much (0.828 v.s. 0.894) and was still satisfactory (Buhl, 2023).

We think the model exhibits fairly good performance on unseen samples from a completely new dataset. Although it may be lower than the performance on the previous testing dataset, it’s noteworthy that the AI-generated voices originate from different models, use distinct methods, feature various speakers, and differ in audio length compared to our training data. Therefore, we believe our model’s performance remains robust.

Here is an example of unseen samples

#read one image and transform
from PIL import Image
fig, ax = plt.subplots(1,2,figsize=(10,5))
image = Image.open('/content/drive/MyDrive/shared-project-folder/unseen/test/0/fake5_10_1.jpg')
ax[0].imshow(image)
test_image, label = transform(image), 0
test_image = test_image.unsqueeze(0)
_,predicted = torch.max(model_resnet(test_image),1)
ax[0].set_title('Predicted: ' + ['Fake','Real'][predicted.item()]+' Label: '+['Fake','Real'][label])

image = Image.open('/content/drive/MyDrive/shared-project-folder/unseen/test/1/speaker1_6_1.jpg')
ax[1].imshow(image)
test_image, label = transform(image), 1
test_image = test_image.unsqueeze(0)
_,predicted = torch.max(model_resnet(test_image),1)
ax[1].set_title('Predicted: ' + ['Fake','Real'][predicted.item()]+' Label: '+['Fake','Real'][label])
plt.show()

The printed 2 spectrograms came from 2 audios which we were unable to classify by hearing them on ourselves, yet the ResNet18 model was able to classify them correctly.

Qualitative explanation

The key reason that even the baseline CNN model achieved a high validation accuracy of 0.98 lay among the spectrograms. We re-visited some spectrograms in the training set from “Deep-Voice” on Kaggle and ASVspoof2019 LA.

real_spectro = os.listdir('/content/drive/MyDrive/shared-project-folder/kaggle/real')
fake_spectro = os.listdir('/content/drive/MyDrive/shared-project-folder/kaggle/fake')

# Display random five real and fake voice spectrogram
np.random.seed(1)
count = 0
fig, axs = plt.subplots(2,5)
for i in np.random.choice(range(len(real_spectro)),5):
 image_path = real_spectro[i]
 image = mpimg.imread(os.path.join('/content/drive/MyDrive/shared-project-folder/kaggle/real',image_path))
 axs[0,count].imshow(image)
 image_path = fake_spectro[i]
 image = mpimg.imread(os.path.join('/content/drive/MyDrive/shared-project-folder/kaggle/fake',image_path))
 axs[1,count].imshow(image)
 axs[0,count].axis('off')
 axs[1,count].axis('off')
 if count == 0:
 axs[0,count].set_title('Kaggle, real voice')
 axs[1,count].set_title('Kaggle, fake voice')
 count += 1

real_spectro = os.listdir('/content/drive/MyDrive/shared-project-folder/LA_Spectrogram/train/1')
fake_spectro = os.listdir('/content/drive/MyDrive/shared-project-folder/LA_Spectrogram/train/0')
np.random.seed(1)
count = 0
fig, axs = plt.subplots(2,5)
for i in np.random.choice(range(len(real_spectro)),5):
 image_path = real_spectro[i]
 image = mpimg.imread(os.path.join('/content/drive/MyDrive/shared-project-folder/LA_Spectrogram/train/1',image_path))
 axs[0,count].imshow(image)
 image_path = fake_spectro[i]
 image = mpimg.imread(os.path.join('/content/drive/MyDrive/shared-project-folder/LA_Spectrogram/train/0',image_path))
 axs[1,count].imshow(image)
 axs[0,count].axis('off')
 axs[1,count].axis('off')
 if count == 0:
 axs[0,count].set_title('ASVspoof, real voice')
 axs[1,count].set_title('ASVspoof, fake voice')
 count += 1

The spectrograms of the fake audios had some obvious visual differences from those of the real audios, including:

certain frequencies being intensified/diminished
different distances between vertical stripes, i.e. different lengths of silent intervals

These differences could be captured by CNN, and even more easily by ResNet which incorporated skip connections. However, when we humans listened to the audios, we could not hear the spectrograms.

Discussion

In our project, we’ve encountered several challenges related to datasets.

Initially, we thought finding the desired datasets would be straightforward.
However, we later found that the number of available datasets was less than expected. This is partly because some datasets only feature non-human voice audio. And some datasets have imbalance between real and fake audio samples. Certain datasets provide only fake audio or, only real audio.
Another challenge is the length of the audio files; most are under 10 seconds, which means we need to preprocess them to make them suitable for training.
Lastly, limiting our search to only English audio reduces the scope of our dataset options. But the question is: could we use non-English audio data for our task? It is a good question that we need to explore.

The important lesson we have learned is that we should be daring when we are facing unfamiliar tasks.

When we came up with the idea to make a fake voice detector, we were concerned with the fact that we haven’t worked on audios during the course.
Using spectrograms, which are images, as the input was a doubtful method because we suspected that spectrograms might not contain enough features of the audios to begin with.
The ResNet18 model turned out to perform well.

It is worth noting that our final model may be vulnerable to adversarial attacks. For instance, it is possible to train a “modifier” which will erase the extra frequencies present in the spectrogram of fake audios, as illustrated in section ‘Unseen Samples’, in order to confuse our model to misclassify fake audios as real.

References

Jordan J. Bird, Ahmad Lotfi. REAL-TIME DETECTION OF AI-GENERATED SPEECH FOR DEEPFAKE VOICE CONVERSION,24 Aug 2023. https://arxiv.org/pdf/2308.12734.pdf
Yamagishi, Junichi; Todisco, Massimiliano; Sahidullah, Md; Delgado, Héctor; Wang, Xin; Evans, Nicolas; Kinnunen, Tomi; Lee, Kong Aik; Vestman, Ville; Nautsch, Andreas. (2019). ASVspoof 2019: The 3rd Automatic Speaker Verification Spoofing and Countermeasures Challenge database, [sound]. University of Edinburgh. The Centre for Speech Technology Research (CSTR). https://doi.org/10.7488/ds/2555.
Rodríguez, Yohanna; Ballesteros L, Dora Maria; Renza, Diego (2019), “Fake voice recordings (Imitation)”, Mendeley Data, V1, doi: 10.17632/ytkv9w92t6.1
Nikolaj Buhl. F1 Score in Machine Learning. Encord, 18 July 2023. https://encord.com/blog/f1-score-in-machine-learning/

Unfair-ToS: A GPT-Based Unfair Term Of Service Detector

Fri, 01 Sep 2023 00:00:00 +0000

Table of Contents

The Github Repo contains the training datasets, source codes, prompts, and other usefull analysis about this project.

Introduction

Problems in Terms of service

Terms of service (ToS) are agreements between service providers and their users. These ToS documents often contain complex legal language that users struggle to understand. Such clauses may violate consumer laws, compromise users’ rights, and raise privacy concerns.

Why LLMs?

The Large Language Model’s (LLM) proven ability to efficiently extract summaries from complex texts makes it ideal for addressing ToS complexities. The Unfair-ToS employs a GPT-based framework to highlight crucial ToS sentences, offer simplified explanations, and evaluate their fairness, also providing reasons for any unfair terms.

CLAUDETTE

Lippi et al. addressed the presence of unfair terms in infrequently read contracts in their paper ’CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Service’. The authors introduced the idea of utilizing machine learning models, including SVM, CNN, and several hybrid models, to identify potential unfair clauses in ToS. While the classification results from this paper are deemed acceptable, there is room for improvement.

Building upon this foundation to adapt to the rapid evolution in the machine learning community, a more recent paper expanded the dataset to include 100 ToS, providing a more comprehensive coverage of service providers. The same annotated mechanism used in the previous paper was applied, and the study achieved improved classification results using Memory-Augmented Neural Networks.

Current Issues

However, in existing approaches, many crucial terms are annotated as fair simply because they do not violate any laws. Users could benefit from being made aware of such terms without having to read through all the fair terms. Furthermore, the utilization of fine-tuned LLMs, a popular approach in legal studies [7], remains unexplored in the context of Unfair Term classification. These issues have prompted our attention to design a more comprehensive model.

Model Architecture

Important Terms Hightlighting

This model leverages GPT-4 through prompts to highlight sentences and generate simplifications for each highlighted sentence from the input, cleaned Terms of Service (ToS). The prompt used in our model has been fine-tuned using 11 ToS documents, comprising approximately 1500 sentences, through prompt engineering. GPT-4 identifies important sentences, and these serve as input to the text classification model.

Unfair Terms Classification

Each sentence undergoes tokenization using HuggingFace’s predefined ‘GPT2’ tokenizer and is then fed into the pre-trained ‘gpt2’ model. This model has been fine-tuned on a dataset of 100 ToS using the same tokenizer and padded to the maximum length. The output of the model falls into one of five classes. The purpose of this model is to assign a label to each highlighted sentence, assisting users in identifying the fairness of sentences.

Model Showcase

Report

Please read our report to learn more about our project motivation, background information, and model evaluation.

Some highlight

Inspired by Yoon Kim’s paper, we implemented a baseline CNN classification model with k1 = 4 and k2 = 4 for fair/unfair classification. However, its F1 score is only 0.193, indicating a bias towards classifying samples into the class with the majority, which is the ‘fair’ class. Consequently, the baseline CNN struggles to accurately identify ‘unfair’ sentences within the samples.

In contrast, our fine-tuned GPT-2 model surpasses the baseline in both metrics. These findings suggest that the GPT-2 model exhibits greater resilience to class imbalance, particularly when ‘fair’ sentences dominate the corpus, as is often the case.

Prompt

Utilizes GPT-4 through prompts to highlight sentences and generate simplifications for each highlighted sentence from the input, cleaned ToS. The prompt used in our model has been fine-tuned using 11 ToS documents, comprising approximately 3300 sentences, through prompt engineering.

We utilized a chain of thought to guide GPT-4 in text highlighting, ensuring that the output aligns with the user group of specific service providers.

### Steps ###
think step by step,
1. who is the service provider? who is its user population?
2. what should be considered as important sentences for users to read? using the definition given below
3. How can you quantify the importance of a sentence using this definition?
*4. What are the 50 most important sentences?

Current Work

We are presently developing a Graphic User Interface to showcase the capabilities of our model. An early beta version is now accessible in the Application folder, enabling users to upload a (txt/csv) file or copy and paste the Terms of Service (ToS) document. This allows the application of our Language Model (LLM) prompts with various models using your personal OPENAI API keys, yielding highlighted text results. The fair/unfair classification feature will be integrated into the GUI in a future release.

To utilize this GUI, kindly clone this repository into your local machine and install any missing packages. Following the completion of all preparations, execute the ‘GUI.py’ file to initiate the application.

We have provided a demo Terms of Service (ToS) document for you to experiment with using our GUI. Please click the “SHOW DEMO” button for more information about this demonstration. If you wish to highlight your own ToS document, you must obtain an OpenAI API key (with sufficient credits based on the length of the ToS and the selected model) and upload it using the SETTING function.

Note: This version is an early beta release. Please use it at your own risk. We are not liable for any costs or information leaks incurred while using our services.

We also welcome any contributions to assist us in completing the front-end design!

Text Classification and Data Anaylsis on Cryptocurrency Related Tweets in PySpark Enviorment

Sat, 01 Jan 2022 00:00:00 +0000

Disclaimer and Background

This project is an improvement of the final project of upper year CS course "Data-Intensive Distributed Analytics" at the University of Waterloo by Hugh Chung , Joe Liang, and Shawn Li. The codes for setting up the Pyspark environments in this project are credited to Ali Abedi, the instructor in Winter 2022.

Data in this project is from the Kaggle post "Bitcoin Tweets" under CC0: Public Domain license. The data includes tweets that have #Bitcoin and #btc hashtags from 2016. Additional information about this dataset can be found here.

Cryptocurrency becomes a popular topic in social media and the financial market. On 30 November 2020, bitcoin hit a new all-time high of $19,860. NLP Analysis on the posts related to cryptocurrency in social media could be an interest area of study.

The goal of this project is to demonstrate the ability to use Pyspark and big data computing in text data analysis and supervised learning: tweets text classification. And using the trained model to construct an automatic hash-tagging system for incoming tweets.

The environment and programming language used in this project mainly focus on Pyspark with its RDD and Data Frame interface. Also, Keras in Tensorflow with Pandas is used to train neural network models.

Pyspark Environment

import shutil, os
if os.path.isdir('CryptoTweets'):
 shutil.rmtree('CryptoTweets')
! git clone https://github.com/JOeOJ520/CryptoTweets.git

To get started, let's initialize Spark.

!apt-get update -qq > /dev/null
!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!wget -q https://downloads.apache.org/spark/spark-2.4.8/spark-2.4.8-bin-hadoop2.7.tgz
!tar xf spark-2.4.8-bin-hadoop2.7.tgz
!pip install -q findspark
!tar -xzf CryptoTweets/sql-data.tgz
# install required packages
!pip install pycountry
!pip install pyecharts

To use Spark SQL and the DataFrame interface, creating a SparkSession.

import os
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = "/content/spark-2.4.8-bin-hadoop2.7"

import findspark
findspark.init()

from pyspark.sql import SparkSession
import random

spark = SparkSession.builder.appName("YourTest").master("local[2]").\
config('spark.ui.port', random.randrange(4000,5000)).config("spark.driver.memory", "9g").getOrCreate()

Data Preprocessing

The bitcoin-tweets.csv contains total 1.09G of tweets regarding bitcoins and crpytocurrecies. The below section requires a kaggle.json for authentication purposes in order to download the file.

from google.colab import files

uploaded = files.upload()

#Upload kaggle account verification
for fn in uploaded.keys():
 print('User uploaded file "{name}" with length {length} bytes'.format(
 name=fn, length=len(uploaded[fn])))

# Then move kaggle.json into the folder where the API expects to find it.
!mkdir -p ~/.kaggle/ && mv kaggle.json ~/.kaggle/ && chmod 600 ~/.kaggle/kaggle.json

#Download dataset
!kaggle datasets download "kaushiksuresh147/bitcoin-tweets"
#unzips
!unzip bitcoin-tweets.zip

Using Pyspark SQL Interface to obtain dataframe from Bitcoin_tweets.csv and first 10 rows of the dataset are showed below.

#Read the csv and construct pyspark dataframe
tweets_raw = spark.read.format("csv").option("header","true").load("Bitcoin_tweets.csv")
tweets_raw = spark.read.csv("./Bitcoin_tweets.csv")
#better visual
tweets_raw.limit(10).toPandas()

	user_name	user_location	user_description	user_created	user_followers	user_friends	user_favourites	user_verified	date	text	hashtags	source	is_retweet
0	DeSota Wilson	Atlanta, GA	Biz Consultant, real estate, fintech, startups...	2009-04-26 20:05:09	8534.0	7605	4838	False	2021-02-10 23:59:04	Blue Ridge Bank shares halted by NYSE after #b...	['bitcoin']	Twitter Web App	False
1	CryptoND	None	😎 BITCOINLIVE is a Dutch platform aimed at inf...	2019-10-17 20:12:10	6769.0	1532	25483	False	2021-02-10 23:58:48	"😎 Today, that's this #Thursday, we will do a ...	#Btc #wallet #security expe… https://t.co/go6...	['Thursday', 'Btc', 'wallet', 'security']	Twitter for Android
2	Tdlmatias	London, England	IM Academy : The best #forex, #SelfEducation, ...	2014-11-10 10:50:37	128.0	332	924	False	2021-02-10 23:54:48	Guys evening, I have read this article about B...	None	Twitter Web App	False
3	Crypto is the future	None	I will post a lot of buying signals for BTC tr...	2019-09-28 16:48:12	625.0	129	14	False	2021-02-10 23:54:33	$BTC A big chance in a billion! Price: \487264...	['Bitcoin', 'FX', 'BTC', 'crypto']	dlvr.it	False
4	Alex Kirchmaier 🇦🇹🇸🇪 #FactsSuperspreader	Europa	Co-founder @RENJERJerky \| Forbes 30Under30 \| I...	None	None	None	None	None	None	None	None	None	None
5	#Bitcoin"	2016-02-03 13:15:55	1249.0	1472	10482	False	2021-02-10 23:54:06	This network is secured by 9 508 nodes as of t...	['BTC']	Twitter Web App	False	None	None
6	ZerrBenz™ ⚔ ✪ 20732	Bkk, Thailand	I'm a cat slave 🐱 Interested in Blockchain · T...	2010-01-12 07:00:04	742.0	716	2444	False	2021-02-10 23:53:30	💹 Trade #Crypto on #Binance	None	None	None
7	📌 Enjoy #Cashback 10% of the Trading fee	None	None	None	None	None	None	None	None	None	None	None	None
8	📌 Sign up link 👉 https://t.co/T4WttWeohc… http...	['Crypto', 'Binance', 'Cashback']	Twitter Web App	False	None	None	None	None	None	None	None	None	None
9	Bitcoin-Bot	Florida, USA	Bot to generate Bitcoin picture as combination...	2019-12-23 16:49:16	131.0	84	5728	False	2021-02-10 23:53:17	<'fire' & 'man'>	None	None	None

The dimension of raw dataset is (row = 11176654, columns = 13). The sample size exceeds the need of our project goal, which might lead to extremely high computational cost.

print((tweets_raw.count(), len(tweets_raw.columns)))

Therefore, we cleans up the dataframe by removing samples with missing value and performs type conversion on multiple columns.

import pyspark.sql.functions as F
from pyspark.sql.types import IntegerType
#Remove all null rows
tweets = spark.sql("SELECT * FROM tweet WHERE user_name != 'null' AND user_description != 'null' \
AND user_location != 'null' AND user_created != 'null' AND user_followers != 'null' AND user_friends != 'null' \
AND user_favourites != 'null' AND user_verified != 'null' AND date != 'null' AND text != 'null' AND hashtags != 'null' \
AND source != 'null' AND is_retweet != 'null'")
#Convert string into datetime for date col
tweets = tweets.withColumn('date', F.to_timestamp('date', 'yyyy-MM-dd'))
tweets = tweets.withColumn('user_created', F.to_timestamp('user_created', 'yyyy-MM-dd'))
#Convert string to Number/Int
tweets = tweets.withColumn("user_followers", tweets["user_followers"].cast(IntegerType()))
tweets = tweets.withColumn("user_friends", tweets["user_friends"].cast(IntegerType()))
tweets = tweets.withColumn("user_favourites", tweets["user_favourites"].cast(IntegerType()))
#Remove all null convertions that occured
tweets = tweets.filter("date is not NULL AND user_created is not NULL \
AND user_followers is not NULL AND user_friends is not NULL")
#Tweet left after cleaning
tweets_left = tweets.count()
#Total tweet
tweets_total = tweets_raw.count()
print("The total number of tweets is: " + str(tweets_total))
print("The total number of tweets after cleaning the data types is " + str(tweets_left))
print("Percentage of tweets removed: ", 1 - (tweets_left / tweets_total))


The total number of tweets is: 11804338
The total number of tweets after cleaning the data types is 497152
Percentage of tweets removed: 0.9578839575755964

Explanatory Data Analysis

Categorical Data

Standardlizing the "user_location" columns into countries using "pycountry" package. Using Pyspark RDDs interface to achieve parallel computing in the calculation of the frequencies of words in "hashtage", "locations", and "source", and "user_name"

from nltk.stem import PorterStemmer
from CryptoTweets.otherstr import *
from CryptoTweets.simple_tokenize import simple_tokenize
# Functions can be found in otherstr.py file in Github
# sources
sor = source(tweets).collect()
# Hashtages
hashs =hashtags(tweets).collect()
# verification
verified = user_verified(tweets).collect()
# user name
name = user_name(tweets).collect()
# Locations
loca = user_location(tweets).collect()

Here is an example of the results from the above calculations. The number indicates the frequency of each category in the dataset.


loca[:10]

[('others', 397175),
 ('United States', 13592),
 ('United Kingdom', 9272),
 ('Canada', 8544),
 ('India', 7791),
 ('Australia', 4415),
 ('Bangladesh', 3545),
 ('South Africa', 3314),
 ('Niger', 3206),
 ('France', 2697)]

import matplotlib.pyplot as plt
# Plots for tweet source
x_t,y_t = zip(*sor)

fig1, ax1 = plt.subplots()
explode = (0.05,0,0,0)
colors = ['#8B0000','#db6777','#e6d1d4','#e8dcde']
ax1.pie(y_t,labels=x_t, colors = colors, autopct='%1.1f%%', startangle=90, pctdistance=0.85, explode = explode)

#draw circle
centre_circle = plt.Circle((0,0),0.70,fc='white')
fig = plt.gcf()
fig.gca().add_artist(centre_circle)

# Equal aspect ratio ensures that pie is drawn as a circle
ax1.axis('equal')
plt.tight_layout()
plt.show()

plt.show()

User Location

# Importing required library
from pyecharts.charts import Bar
from pyecharts import options as opts


# Obtaining x and y axis from Location lists
x_hash,y_hash = zip(*loca)
bar = (
 Bar(init_opts=opts.InitOpts())
 .add_xaxis(x_hash[1:11])
 .add_yaxis("Frequency",y_hash[1:11])
 .set_global_opts(title_opts=opts.TitleOpts(title="Top 10 User Location", subtitle="standardization and removed others"))
)
bar.render_notebook()

plt.xticks(
 rotation=45,
 horizontalalignment='right',
 fontweight='light',
 fontsize='x-large'
)

plt.bar(x_hash[1:10], y_hash[1:10])
plt.show()

User Verification

# Plots for verified
x_t,y_t = zip(*verified)

fig1, ax1 = plt.subplots()
explode = (0, 0)
colors = ['#6067e0','#fc0303']
ax1.pie(y_t,labels=x_t, colors = colors, explode = explode, autopct='%1.1f%%',
 shadow=True, startangle=90)
ax1.axis('equal') # Equal aspect ratio ensures that pie is drawn as a circle.
#draw circle
centre_circle = plt.Circle((0,0),0.70,fc='white')
fig = plt.gcf()
fig.gca().add_artist(centre_circle)

# Equal aspect ratio ensures that pie is drawn as a circle
ax1.axis('equal')
plt.tight_layout()
plt.show()

plt.show()

Hashtag (Response Variable)

# Counting the frequency of each hashtags
x_hash,y_hash = zip(*hashs)
bar = (
 Bar(init_opts=opts.InitOpts())
 .add_xaxis(x_hash[1:11])
 .add_yaxis("Frequency",y_hash[1:11])
 .set_global_opts(title_opts=opts.TitleOpts(title="Top 10 Hashtags in the Tweets"))
)
bar.render_notebook()

 <div id="8d97781c87bf495ca55aa7e8b30493a5" style="width:900px; height:500px;"></div>

The non-interactive plot when the above interactive plot fail to load

plt.xticks(
 rotation=45,
 horizontalalignment='right',
 fontweight='light',
 fontsize='x-large'
)

plt.bar(x_hash[1:10], y_hash[1:10])
plt.show()

We decides to use four of the most frequent hashtags and "bitcoin" as our five response variables for text classification in supervised learning. Therefore, the goal is to classify each tweet into one of the five categories using the trained model in the future sections.

Numerical Variables

Performing calculation on numerical variables in the dataset, such as "Post date", "user created date", "Number of followers", and others. Obtaining the frequency of possible values in the samples.

Number of tweets in recent two years

date_count = tweets.select("date").rdd.flatMap(lambda row: [(row[0], 1)]).\
 reduceByKey(lambda x,y: x+y).sortBy(lambda x: x[0]).collect()
date_x, date_y = zip(*date_count)
plt.figure(figsize=(15, 5))
plt.plot(date_x, date_y)
plt.title('Total tweets by Date')
plt.xlabel('Date')
plt.ylabel('Number of Tweets')
plt.show()

We also performed the time series decomposition on the samples to check possible seasonal patterns and trends.

import pandas as pd
from statsmodels.tsa.seasonal import seasonal_decompose
series = pd.DataFrame(date_count)
result = seasonal_decompose(series[1], model='additive', freq=12)
result.plot()
plt.show()

There is an increasing trend at the beginning of the plot with a very low residual. However, there is no clear pattern after 07/2021.

} Account Created Date

Applying Log transformation to the number of account.

from numpy import log as ln
created_count = tweets.select("user_created").rdd.flatMap(lambda row: [(row[0], 1)]).\
reduceByKey(lambda x,y: x+y).sortBy(lambda x: x[0]).collect()
#Convert list of tuple into two lists
date_x, date_y = zip(*created_count )
plt.figure(figsize=(15, 5))
plt.plot(date_x, ln(date_y))
plt.title('Total Account Created by Date')
plt.xlabel('Date')
plt.ylabel('Number of Account')
plt.show()

series = pd.DataFrame(created_count)
result = seasonal_decompose(ln(series[1]), model='additive', freq=12)
result.plot()
plt.show()

There is no significant evidence in the time series decomposition to support the existence of a seasonal pattern in the samples.

Number of followers of each tweet user

#Get the count of number of friends for the accounts
def checker(x):
 if x[0] < 50:
 return ('1',x[1])
 elif x[0] < 100:
 return ('2',x[1])
 elif x[0] < 200:
 return ('3',x[1])
 elif x[0] < 1000:
 return ('4',x[1])
 else:
 return ('5',x[1])
friends = tweets.select("user_friends").rdd.flatMap(lambda row: [(row[0], 1)])\
 .map(lambda x: checker(x)).reduceByKey(lambda x,y: x+y).collect()
#Convert list of tuple into two lists
friends_x, friends_y = zip(*friends)
fig1, ax1 = plt.subplots()
colors = ['#ffbaba','#ff7b7b','#ff5252','#ff0000','#a70000']
ax1.pie(friends_y,labels=['< 50','50-100','100-200','200-1000','>1000'], \
 autopct='%1.1f%%', colors = colors, startangle=90, pctdistance=0.85)

#draw circle
centre_circle = plt.Circle((0,0),0.70,fc='white')
fig = plt.gcf()
fig.gca().add_artist(centre_circle)

# Equal aspect ratio ensures that pie is drawn as a circle
ax1.axis('equal')

plt.tight_layout()
plt.title('Total number of followers')
plt.show()

Data cleaning for text classification

After data analysis, we found that some variables can be cleaned into a more suitable format for machine learning. For example, the response variables "hashtag" can be eliminated into five categories. And the variable "sources" is a categorical variable with four levels. We converted it into indicator variables.

from pyspark.sql.functions import when
from pyspark.sql.functions import monotonically_increasing_id
from pyspark.sql.functions import udf
from pyspark.sql.functions import year,month
from pyspark.sql import functions as F

# Cleaning response variables
tweet_ml = tweets.withColumn('hashtags', when(tweets.hashtags.contains("dog"),'Dogecoin').\
 when(tweets.hashtags.contains("eth"),'Etherenum').\
 when(tweets.hashtags.contains("bnb"),"binance").\
 when(tweets.hashtags.contains("bin"),"binance").\
 when(tweets.hashtags.contains("crypto"),'Cryptocurrency').\
 when(tweets.hashtags.contains("btc"),'Bitcoin').\
 otherwise('other'))

# Assigning unique_id to each row
unique_id = monotonically_increasing_id()
tweet_ml = tweet_ml.select("*").withColumn("id", unique_id)

# Cleaning locations
import pycountry
def get_country(x):
 for country in pycountry.countries:
 if country.name in x:
 return country.name
 return "others"
get_countryudf = udf(lambda z: get_country(z))

#Creating indicator variables for categorical variables data
tweet_ml = tweet_ml.na.drop().withColumn("user_location", get_countryudf("user_location"))\
.withColumn('user_verified', when(tweets.user_verified.contains("True"),1).otherwise(0))\
.withColumn("source_Iphone", when(tweets.source.contains("iPhone"),1).otherwise(0))\
.withColumn("source_Web", when(tweets.source.contains("Web"),1).otherwise(0))\
.withColumn("source_Android", when(tweets.source.contains("Android"),1).otherwise(0))\
.withColumn("post_year",year(tweet_ml.date))\
.withColumn("post_month",month(tweet_ml.date))\
.withColumn("created_year",year(tweet_ml.user_created))\
.withColumn("created_month",month(tweet_ml.user_created)).drop('is_retweet','user_name',"user_created","date","source")
# better visual
tweet_ml.limit(5).toPandas()

	user_location	user_description	user_followers	user_friends	user_favourites	text	hashtags	id	source_Iphone	source_Web	post_year	post_month	created_year	created_month
0	others	Biz Consultant, real estate, fintech, startups...	8534	7605	4838	Blue Ridge Bank shares halted by NYSE after #b...	other	0	0	1	2021	2	2009	4
1	others	Biz Consultant, real estate, fintech, startups...	8534	7605	4838	.@Tesla’s #bitcoin investment is revolutionary...	Cryptocurrency	1	0	1	2021	2	2009	4
2	others	Persistent. to the extreme... #FREEPALESTINE #...	1159	2185	30852	Annnd #btc #Bitcoin is headed even higher now....	Bitcoin	2	0	1	2021	2	2009	1
3	others	#Bitcoin	4	32	139	Buy #Bitcoin with 5% LIFETIME cashback on fees...	Cryptocurrency	3	0	1	2021	2	2010	7
4	others	Biz Consultant, real estate, fintech, startups...	8534	7605	4838	#Bitcoin institutional demand accelerates in 2...	Cryptocurrency	4	0	1	2021	2	2009	4
5	others	CEO & PRESIDENT SG GROUP	62	288	2656	#Bitcoin #BTC #ADA #DOT Mastercard Will Let Me...	other	5	1	0	2021	2	2009	6
6	others	Biz Consultant, real estate, fintech, startups...	8534	7605	4838	After @Tesla: @Twitter considers adding #bitco...	other	6	0	1	2021	2	2009	4
7	Portugal	#bitcoin Entrepreneur, Master in Communication...	872	158	1080	#BTC/USD 4H. #Bitcoin consolidating between su...	other	7	0	0	2021	2	2020	9
8	others	Biz Consultant, real estate, fintech, startups...	8534	7605	4838	The @Grayscale #Bitcoin Trust: What it is and ...	other	8	0	1	2021	2	2009	4
9	others	One bet every day. Join our team and become pa...	2019	104	71	We accept #Bitcoin, #BitcoinCash #Litecoin and...	Dogecoin	9	0	1	2021	2	2014	12

Nature Language Processing on user descriptions and tweets: Tokens

The user descriptions and tweets can be considered natural human language. They both share some same characteristics: long sentences, emojis, and containing some unwanted symbols.

To analyze these two variables, we first convert all texts into bags of words, including stemming, converting to lowercase, and deleting all possible stopwords.

Then we calculate the Frequency for each words and selected the highest 20 words to be included in our text classification model.

from CryptoTweets.simple_tokenize import simple_tokenize
from nltk.stem import PorterStemmer
import re
# Top 20 words
n = 20
#Take the text
tweets_text = tweet_ml.select("text")
#Take the user description
tweets_ud = tweet_ml.select("user_description")
# Stemming using Porter Stemmer
st = PorterStemmer()
#stop words
with open('CryptoTweets/CommonEnglishWord.txt') as f:
 lines = f.readlines()
 lst = list(map(lambda x: x[0:len(x)-1].lower(),lines))
 lst.append('')
 lst.append('-')
 lst.append("it's")
 lst.append("going")
 lst.append("it’s")
 lst.append("via")
 lst.append("|")
 lst.append("&")
 lst.append("/")
 lst.append('•')
 lst.append('http')
# Remove emoji since it beyonds the scope of this scope
def deEmojify(text):
 regrex_pattern = re.compile(pattern = "["
 u"\U0001F600-\U0001F64F" # emoticons
 u"\U0001F300-\U0001F5FF" # symbols & pictographs
 u"\U0001F680-\U0001F6FF" # transport & map symbols
 u"\U0001F1E0-\U0001F1FF" # flags (iOS)
 "]+", flags = re.UNICODE)
 return regrex_pattern.sub(r'',text)
# Bag of words, stemming, lowercase, stop words
rddtext = tweets_text.rdd.flatMap(lambda x: simple_tokenize(deEmojify(x[0]))).\
 map(lambda x: st.stem(x)).filter(lambda x: x not in lst).filter(lambda x: len(x) > 1).\
 map(lambda x: (x.lower(),1)).reduceByKey(lambda x,y: x+y).sortBy(lambda x: x[1],ascending=False).cache()

Then we created variables for each of top 20 words. The value indicate the Term Frequency of each words in current text. The following table shows the resulted variables of first ten samples.

# Calculate the frequency and return the result as tuple
def calcfreq(t):
 wordtup = t[1]
 wordlist = []
 tweet = t[0].lower()
 for i in wordtup:
 wordlist.append(list(i))
 for i in wordlist:
 if i[0] in tweet:
 i[1] += 1
 result = [t[0]]
 for i in wordlist:
 result.append(i[1])
 return tuple(result)

reinit_list = rddtext.map(lambda x: (x[0], 0)).take(n)
most_frequent_tweet = rddtext.take(n)
words, freq = zip(*most_frequent_tweet)
words = list(words)
words.insert(0,'text')
reinit_rdd = tweets_text.rdd.map(lambda x:x[0]).map(lambda x: (x, reinit_list))
calc = reinit_rdd.map(lambda x: calcfreq(x))
table_tweet = calc.toDF(words)
table_tweet.limit(5).toPandas()

	text	bitcoin	co	btc	crypto	cryptocurr	...	ha	wa	invest
0	Blue Ridge Bank shares halted by NYSE after #b...	1	1	0	0	0	...	1	0	0
1	.@Tesla’s #bitcoin investment is revolutionary...	1	1	0	1	0	...	0	0	1
2	Annnd #btc #Bitcoin is headed even higher now....	1	1	1	0	0	...	1	0	0
3	Buy #Bitcoin with 5% LIFETIME cashback on fees...	1	1	0	1	1	...	1	0	0
4	#Bitcoin institutional demand accelerates in 2...	1	1	1	1	1	...	0	1	0

5 rows × 21 columns

The follow plot shows the frequency distribution of top 20 words

x_t,y_t = zip(*most_frequent_tweet)
bar = (
 Bar(init_opts=opts.InitOpts())
 .add_xaxis(x_t)
 .add_yaxis("Frequency",y_t)
 .set_global_opts(title_opts=opts.TitleOpts(title="Top 20 words in the Tweets"))
)
bar.render_notebook()

 <div id="0c81b88fcab84b0ebc1683a83d5ca866" style="width:900px; height:500px;"></div>

The non-interactive plot when the above interactive plot fail to load

plt.xticks(
 rotation=45,
 horizontalalignment='right',
 fontweight='light',
 fontsize='x-large'
)
plt.bar(x_t, y_t)
plt.show()

table_tweet = table_tweet.select("*").withColumn("idtweet", unique_id)
tweet_ml = tweet_ml.join(table_tweet,tweet_ml.id == table_tweet.idtweet,'inner').drop('text','_1','idtweet')

Similiar preparation and variable creations for variable "user description"

udrdd = tweets_ud.rdd.flatMap(lambda x: simple_tokenize(deEmojify(x[0]))).\
 map(lambda x: st.stem(x)).filter(lambda x: x not in lst).filter(lambda x: len(x) > 1).\
 map(lambda x: (x.lower(),1)).reduceByKey(lambda x,y: x+y).sortBy(lambda x: x[1],ascending=False).cache()
reinit_udlist = udrdd.map(lambda x: (x[0],0)).take(n)
most_frequent_ud = udrdd.take(n)
words, freq = zip(*most_frequent_ud)
words = list(words)
words.insert(0,'text')
reinit_udrdd = tweets_ud.rdd.map(lambda x:x[0]).map(lambda x: (x, reinit_udlist))
calcud = reinit_udrdd.map(lambda x: calcfreq(x))
table_ud = calcud.toDF(words)
table_ud = table_ud.select("*").withColumn("idud", unique_id)
table_ud = table_ud.select([F.col(c).alias("ud"+c) for c in table_ud.columns])
tweet_ml = tweet_ml.join(table_ud,tweet_ml.id == table_ud.udidud,'inner').drop('udtext','ud_1','udidud').cache()

x_t,y_t = zip(*most_frequent_ud)
bar = (
 Bar(init_opts=opts.InitOpts())
 .add_xaxis(x_t)
 .add_yaxis("Frequency",y_t)
 .set_global_opts(title_opts=opts.TitleOpts(title="Top 20 words in the User Descriptions"))
)
bar.render_notebook()

 <div id="381439a01c6c488787d31b7bed284317" style="width:900px; height:500px;"></div>

The non-interactive plot when the above interactive plot fail to load

plt.xticks(
 rotation=45,
 horizontalalignment='right',
 fontweight='light',
 fontsize='x-large'
)
plt.bar(x_t, y_t)
plt.show()

Here are the first five samples of the dataset after applying natural language processing to tweets and user descriptions.

We can also calculate the TF-IDF to vectorize the top 20 words in each sample. Compare to frequency, TF-IDF has the advantage by assigning a larger weight to words that appear less in the documents. It can be a future improvement.

tweet_ml = tweet_ml.drop('user_description')
tweet_ml.limit(5).toPandas()

	user_location	user_followers	user_friends	user_favourites	hashtags	id	source_Iphone	...	udinvestor	udtrader	udtweet	udinvest	uddoge
0	others	8	0	49	Dogecoin	26	1	...	0	0	1	0	0
1	others	94	189	753	other	29	1	...	0	0	0	0	1
2	others	5366	927	34484	other	474	1	...	1	1	0	1	0
3	United States	68	84	427	other	964	1	...	0	0	0	0	0
4	others	275	789	3654	other	1677	1	...	0	0	0	0	0

5 rows × 54 columns

Training, Testing, and Validation Dataset

Since not all samples belong to these five response variables (hashtags = 'other' in above table), we decided to use these un-classified samples as our testing dataset to demonstrate the outcome of our training model in future sections.

tweet_train = tweet_ml.filter(~tweet_ml.hashtags.contains('other')).cache()
tweet_test = tweet_ml.filter(tweet_ml.hashtags.contains('other')).cache()

Here is the final distribution of the response variables in the training set in text classifications.

hashtag_ml =hashtags(tweet_train).collect()
# Cleaned Hashtag
x_hash = []
y_hash = []
for i in hashtag_ml:
 x_hash.append(i[0])
 y_hash.append(i[1])

plt.xticks(
 rotation=45,
 horizontalalignment='right',
 fontweight='light',
 fontsize='x-large'
)

plt.bar(x_hash[1:10], y_hash[1:10])
plt.show()

Then spliting training dataset into training data and validation data (80% and 20%) to test the performance of the training model.

Multinomial Regression Model

Setting up the Pyspark Machine Learning environments. Removing user locations since it has too many levels which leads to extremely high computational cost.

from pyspark.ml.classification import LogisticRegression
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.feature import StringIndexer

# Y-variables
stringIndexer = StringIndexer(inputCol="hashtags", outputCol="label")
model = stringIndexer.fit(tweet_train)
td = model.transform(tweet_train)

# X-variables
assembler = VectorAssembler(outputCol= "features")\
.setInputCols(tweet_train.drop('hashtags','id','user_location').columns)

# Setting up Multinomial Logistic Regression
lr = LogisticRegression(maxIter=10,family="multinomial")
assembler_df = assembler.transform(td)
train, validation = assembler_df.randomSplit([0.8,0.2],2022)

Fitted a Multinomial Logestic Regression model using training data

lrModel = lr.fit(train)

Tuning the model and checking the performance using validation data

predictions = lrModel.transform(validation)
accuracy = predictions.filter(predictions.label == predictions.prediction).count()/float(predictions.count())
print("The accuracy of prediction in Validation Data",accuracy)

The accuracy of prediction in Validation Data 0.6524463640869503

Random Forest

Using the training set to train a Random Forest with 100 total trees, and number of sqrt(col) variables to choose in each trees.

from pyspark.ml.classification import RandomForestClassifier
rf = RandomForestClassifier(featuresCol = 'features', labelCol = 'label',numTrees=100)
rfModel = rf.fit(train)

Checking the performance using validation data

predictions = rfModel.transform(validation)
accuracy = predictions.filter(predictions.label == predictions.prediction).count()/float(predictions.count())
print("The accuracy of prediction in Validation Data",accuracy)

The accuracy of prediction in Validation Data 0.8449691991786448

Neural Network with Keras API

Converting to Pandas Dataframe for better compatibility with Keras Packages. And creating dummy variables for response variables "hashtags" since the neural network requires int variables for calculation.

train_df = tweet_train.drop('id').toPandas()
response = train_df.pop('hashtags')
train_df = train_df.drop(columns= 'user_location')
response = pd.get_dummies(response)

Setting up the Keras environments and a neural network model with two hidden layers, have 128 and 256 node. Setting up the Relu activation functions for non-linearity, and softmax output function for multi-class classification.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

inputs = keras.Input(shape=(51,))
x = layers.Dense(128, activation="relu", name="dense_1")(inputs)
x = layers.Dense(256, activation="relu", name="dense_2")(x)
outputs = layers.Dense(5, activation="softmax", name="classification")(x)

model = keras.Model(inputs=inputs, outputs=outputs)
model.summary()

Model: "model_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_2 (InputLayer) [(None, 51)] 0
dense_1 (Dense) (None, 128) 6656
dense_2 (Dense) (None, 256) 33024
classification (Dense) (None, 5) 1285
=================================================================
Total params: 40,965
Trainable params: 40,965
Non-trainable params: 0
_________________________________________________________________

The above shows the structures of our final model. And fitting it with training data and tunning it with validation splits

model.compile(optimizer='adam',
 loss='categorical_crossentropy',
 metrics = 'accuracy')
checkpoint_filepath = '/tmp/checkpoint'
model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(
 filepath=checkpoint_filepath,
 save_weights_only=True,
 monitor='loss',
 mode='max',
 save_best_only=True)
model.fit(train_df, response, batch_size=32, epochs=20, validation_split=0.2, callbacks=[model_checkpoint_callback])

Epoch 1/20
3548/3548 [==============================] - 14s 4ms/step - loss: 116.8460 - accuracy: 0.4272 - val_loss: 21.8834 - val_accuracy: 0.4696
Epoch 2/20
3548/3548 [==============================] - 11s 3ms/step - loss: 11.9903 - accuracy: 0.4798 - val_loss: 1.3669 - val_accuracy: 0.4488
Epoch 3/20
3548/3548 [==============================] - 11s 3ms/step - loss: 1.2508 - accuracy: 0.4696 - val_loss: 1.6729 - val_accuracy: 0.4723
Epoch 4/20
3548/3548 [==============================] - 11s 3ms/step - loss: 1.0349 - accuracy: 0.5547 - val_loss: 0.8407 - val_accuracy: 0.6537
Epoch 5/20
3548/3548 [==============================] - 13s 4ms/step - loss: 0.7887 - accuracy: 0.6968 - val_loss: 0.7531 - val_accuracy: 0.7269
Epoch 6/20
3548/3548 [==============================] - 11s 3ms/step - loss: 0.7540 - accuracy: 0.7144 - val_loss: 0.8589 - val_accuracy: 0.6768
Epoch 7/20
3548/3548 [==============================] - 11s 3ms/step - loss: 0.7876 - accuracy: 0.7020 - val_loss: 0.7817 - val_accuracy: 0.6801
Epoch 8/20
3548/3548 [==============================] - 11s 3ms/step - loss: 0.7321 - accuracy: 0.7294 - val_loss: 0.6161 - val_accuracy: 0.7643
Epoch 9/20
3548/3548 [==============================] - 11s 3ms/step - loss: 0.9909 - accuracy: 0.6766 - val_loss: 0.9638 - val_accuracy: 0.6175
Epoch 10/20
3548/3548 [==============================] - 10s 3ms/step - loss: 0.8489 - accuracy: 0.6843 - val_loss: 0.7968 - val_accuracy: 0.6997
Epoch 11/20
3548/3548 [==============================] - 10s 3ms/step - loss: 0.8047 - accuracy: 0.7123 - val_loss: 0.8172 - val_accuracy: 0.6729
Epoch 12/20
3548/3548 [==============================] - 11s 3ms/step - loss: 0.7652 - accuracy: 0.7191 - val_loss: 1.0272 - val_accuracy: 0.5862
Epoch 13/20
3548/3548 [==============================] - 11s 3ms/step - loss: 0.8149 - accuracy: 0.6901 - val_loss: 0.6500 - val_accuracy: 0.7623
Epoch 14/20
3548/3548 [==============================] - 11s 3ms/step - loss: 0.8350 - accuracy: 0.6905 - val_loss: 0.9390 - val_accuracy: 0.6809
Epoch 15/20
3548/3548 [==============================] - 11s 3ms/step - loss: 0.7641 - accuracy: 0.7348 - val_loss: 0.6914 - val_accuracy: 0.7622
Epoch 16/20
3548/3548 [==============================] - 10s 3ms/step - loss: 0.7264 - accuracy: 0.7385 - val_loss: 0.6975 - val_accuracy: 0.7328
Epoch 17/20
3548/3548 [==============================] - 10s 3ms/step - loss: 0.7544 - accuracy: 0.7320 - val_loss: 0.9025 - val_accuracy: 0.7154
Epoch 18/20
3548/3548 [==============================] - 11s 3ms/step - loss: 0.7749 - accuracy: 0.7043 - val_loss: 0.6575 - val_accuracy: 0.7617
Epoch 19/20
3548/3548 [==============================] - 11s 3ms/step - loss: 0.7928 - accuracy: 0.7032 - val_loss: 0.9805 - val_accuracy: 0.6677
Epoch 20/20
3548/3548 [==============================] - 10s 3ms/step - loss: 0.6914 - accuracy: 0.7542 - val_loss: 0.6055 - val_accuracy: 0.7811
<keras.callbacks.History at 0x7f1ebf43ec10>

The above plot demonstrates the training process of the neural network model in each iteration. To avoid overfitting, we need to use the model with the lowest validated loss (val_loss) in the above result. This model is stored in memory by checkpoint functions and is callable for future usage.

Prediction

We will use the testing dataset to demonstrate the outcome of three trained models and how auto-hashtaging system works in incoming tweets.

Logestic Regression

from pyspark.ml.feature import IndexToString
test = assembler.transform(tweet_test)
test_prediction = lrModel.transform(test)
backtoshash = IndexToString(inputCol="prediction", outputCol="hashes",labels=['Cryptocurrency','Bitcoin','Dogecoin','Etherenum','binance'])
test_prediction = backtoshash.transform(test_prediction)
test_prediction.select('id','probability').show(10,False)

+----+------------------------------------------------------------------------------------------------------+
|id |probability |
+----+------------------------------------------------------------------------------------------------------+
|29 |[0.3077313892095184,0.371686180208301,0.16157765845611455,0.15900331096889617,1.4611571697825333E-6] |
|474 |[0.33327388585249207,0.3741639190318949,0.1714864279527474,0.12107430591892694,1.4612439387007495E-6] |
|964 |[0.3913783911510319,0.3027252404883092,0.17921642595022608,0.1266785265102223,1.4159002104801821E-6] |
|1677|[0.33295682110539726,0.3273996767104981,0.16134718703164241,0.17829488835432508,1.4267981371979524E-6]|
|1950|[0.3600914851863604,0.24751148870741563,0.15465675775777693,0.23773885594117347,1.4124072734451633E-6]|
|2040|[0.42485527012649427,0.26050835049863025,0.16349387679216867,0.1511411729427666,1.3296399402649282E-6]|
|2214|[0.4419353500007448,0.2527123390106577,0.16882974288092872,0.13652119501290258,1.3730947663475665E-6] |
|2453|[0.4589381243296783,0.14606804491493697,0.2937284738352546,0.10126408940553218,1.2675145980027649E-6] |
|2509|[0.3321477952505388,0.37096013204945966,0.16393197503583595,0.13295870829424924,1.389369916457664E-6] |
|2529|[0.3048276563348717,0.23617205430574892,0.2686116148297133,0.19038718971765053,1.484812015595407E-6] |
+----+------------------------------------------------------------------------------------------------------+
only showing top 10 rows

The hashtags with the highest probabilities will be the classified categories for the corresponding samples (optimal Bayes)

test_prediction.select('id','hashes').show(10,False)

+----+--------------+
|id |hashes |
+----+--------------+
|29 |Bitcoin |
|474 |Bitcoin |
|964 |Cryptocurrency|
|1677|Cryptocurrency|
|1950|Cryptocurrency|
|2040|Cryptocurrency|
|2214|Cryptocurrency|
|2453|Cryptocurrency|
|2509|Bitcoin |
|2529|Cryptocurrency|
+----+--------------+
only showing top 10 rows

Random Forest Similiar for Random Forest model and Neural Network Model

test_prediction = rfModel.transform(test)
backtoshash = IndexToString(inputCol="prediction", outputCol="hashes",labels=['Cryptocurrency','Bitcoin','Dogecoin','Etherenum','binance'])
test_prediction = backtoshash.transform(test_prediction)
test_prediction.select('id','probability').show(10,False)

test_prediction.select('id','hashes').show(10,False)

+----+-----------------------------------------------------------------------------------------------------+
|id |probability |
+----+-----------------------------------------------------------------------------------------------------+
|29 |[0.11408553240964694,0.7291978857094307,0.06664959177602166,0.055024281883508205,0.03504270822139252]|
|474 |[0.1652564305753442,0.6331858397395639,0.08087177629844432,0.037126306578410845,0.08355964680823667] |
|964 |[0.3305776828116616,0.36432798930213495,0.14284028703120938,0.06453066609337645,0.0977233747616177] |
|1677|[0.31612928358927095,0.4227445340598757,0.11989082364177438,0.05751233645147351,0.08372302225760536] |
|1950|[0.08574619059640502,0.16340390330095428,0.03166511566726569,0.7003261461305691,0.01885864430480592] |
|2040|[0.5943068155443776,0.2726191016966887,0.07905983918076226,0.03180926486801036,0.022204978710161125] |
|2214|[0.6667520715299391,0.1644187178079488,0.10068119920639075,0.033609698746054184,0.03453831270966719] |
|2453|[0.4000689206511194,0.02974922008303281,0.5285874094943018,0.028858445070719826,0.012736004700826165]|
|2509|[0.1381026785114682,0.739044371954166,0.06888786575378644,0.027324394280704465,0.026640689499874904] |
|2529|[0.19059071422634632,0.15944406647246737,0.554794414228412,0.05290261631835627,0.042268188754417985] |
+----+-----------------------------------------------------------------------------------------------------+
only showing top 10 rows
+----+--------------+
|id |hashes |
+----+--------------+
|29 |Bitcoin |
|474 |Bitcoin |
|964 |Bitcoin |
|1677|Bitcoin |
|1950|Etherenum |
|2040|Cryptocurrency|
|2214|Cryptocurrency|
|2453|Dogecoin |
|2509|Bitcoin |
|2529|Dogecoin |
+----+--------------+
only showing top 10 rows

Neural Network

test_df = tweet_test.drop('id').toPandas()
response = test_df.pop('hashtags')
test_df = test_df.drop(columns= 'user_location')
model.load_weights(checkpoint_filepath)
prediction = model.predict(test_df)
prediction[:10]

array([[2.35893115e-01, 3.70337307e-01, 8.22583735e-02, 3.05431724e-01,
6.07949682e-03],
[1.47559753e-04, 9.99817193e-01, 1.51092713e-12, 3.53277137e-05,
2.90755020e-22],
[3.74551356e-01, 2.91717589e-01, 1.80127278e-01, 1.19809344e-01,
3.37944217e-02],
[8.34352300e-02, 4.13003623e-01, 1.82284534e-01, 2.89828300e-01,
3.14484239e-02],
[3.59519780e-01, 4.05398160e-01, 1.29304364e-01, 1.38947032e-02,
9.18831453e-02],
[0.00000000e+00, 9.99999642e-01, 0.00000000e+00, 4.06629681e-07,
0.00000000e+00],
[9.14618373e-02, 5.94880283e-01, 1.25181660e-01, 1.41764238e-01,
4.67120372e-02],
[3.66317110e-09, 9.95616674e-01, 1.36035316e-08, 2.10694573e-03,
2.27643130e-03],
[5.13970926e-02, 3.42191756e-01, 3.40411484e-01, 9.60622579e-02,
1.69937387e-01],
[1.38813183e-01, 2.94758379e-01, 1.67302951e-01, 3.93695772e-01,
5.42974332e-03]], dtype=float32)

import numpy as np
label = ['Bitcoin','Cryptocurrency','Dogecoin','Etherenum','binance']
hashtags = []
for prob in prediction:
 index_max = np.argmax(prob)
 hashtags.append(label[index_max])
hashtags[:10]

['Cryptocurrency',
'Cryptocurrency',
'Bitcoin',
'Cryptocurrency',
'Cryptocurrency',
'Cryptocurrency',
'Cryptocurrency',
'Cryptocurrency',
'Cryptocurrency',
'Etherenum']

Conclusion and Possible Improvement

This project demonstrates data analysis of tweets and related information under Pyspark environments. And performing a simple text classification using three popular supervised learning models: logistic regression, Random Forest, and ANN. The Random forest model achieves 84% accuracy in the hashtag classification of the validation dataset. Therefore, the prediction can be an important metrics for an automatic tagging system for new tweets.

There are alterative choices of machine learning models for text classification in this project. For exmaple, the vectorized version of the neural network, support vector machine, KNN, and others. Some of models might have a better performance than the above models. These can be a possible improvement of this project in future developments.

Also, Twitter API provides more possibilities for data mining, such as but not limited to streaming, recent search, and particular user search. Therefore, the Kaggle dataset in this project can be replace with other sources to improve the performance.

Tranditional Statistical Learning: Classification in Self-Assessed Financial Health Status

Sat, 01 Jan 2022 00:00:00 +0000

Disclaimer

This project is an improvement of the final project of the upper-year Statistic course “STAT441: Statistical Learning - Classification” at the University of Waterloo by Bolun Cui and Joe Liang.

Video

A video explaination about this project can be found here. (Note: The video was made for the university course project, some parts in the video might not be matched with the file)

Background

The dataset in this project corresponds to the responses in the German General Social Survey (ALLBUS) between 2005 and 2019. The target variable for machine learning is the last variable “health”. It is an ordinal variable with five categories from 1 to 5 and represents the “self-asset financial health” of each survey response.

There are two parts of the dataset, ”train.csv" and “test.csv”. the samples in ”train.csv" include “health” variables, which are used for model training. And “test.csv” does not have the “health” variable. The goal of this project is to train a classification model using the “train.csv” to classify survey responses in “test.csv” into one of the financial health categories.

Highlights

Compelete documentation can be found in the “Supervised Learning code.rmd” file

Exploratory Data Analysis

Outlier anaylsis

Target variables distribution anaylsis and normalization

Feature Engineering

During analysis, based on our domain knowledge, we derived a new x-variable: the average living space in m2 per person in the household.

Random Foresting

Out of Bag (OOB) samples tuning for number of variables to choice and number of trees

Importance of variables from OOB (randomly mix each variables to test the decrease in accuracy)

Tracing the perfomance of different number of trees

Neural Network

Pipline implentation of two hidden layers neural network

Tuning Epochs (number of iteration) to balance bias and vairance tradeoff

Number Nodes and Layer tuning with validation cross entropy

Performance of the Model

Enviorment

This project uses R with R markdown for better visualization. Please visit the official websites for documentation and installation of R, and R Markdown. R studio is recommended to open the .rmd file.

The required packages to excuate the code in .rmd file are listed below and can be installed in CRAN using

install.package("package_name")

in R or R studio.

randomForest: a comprehensive package for Random Forest Model training
caret: a machine learning platform with many integrated features, such as cross-validation
fastDummies: a package allows you to convert categorical variables into indicator (Dummy) variables
Keras: a comprehensive package under Tensorflow for Neural Network (Tensorflow installation is required)
gbm: the Generalized Boosting Model is supported
nnet: the Multinomial logistic regression model is supported

project | Joe Liang

ProtectYourVoice: Detacting AI Generated Voice using Deep Learning

Introduction

Methodology

Current appoarch

Pipline

Import packages

Dataset

“DEEP-VOICE: DeepFake Voice Recognition” from kaggle

Download and process dataset

Converting real voice

Convert fake voice

Some examples

Train/Test splitting and regrouping samples into Pytorch ImageFolder structure

ASVspoof 2019 from DataShare

Basic statistic of dataset

Combining samples

Converting into spectrogram

Unseen samples for model evaluation

Model training and eveluation

Baseline Model

Training

Evaluation

Transfer Learning: Pre-train Res-Net

Model Training and Fine-tuning

Unseen samples

Qualitative explanation

Discussion

References

Unfair-ToS: A GPT-Based Unfair Term Of Service Detector

Introduction

Problems in Terms of service

Why LLMs?

Related Works

CLAUDETTE

Current Issues

Model Architecture

Important Terms Hightlighting

Unfair Terms Classification

Model Showcase

Report

Some highlight

Prompt

Current Work

Text Classification and Data Anaylsis on Cryptocurrency Related Tweets in PySpark Enviorment

Disclaimer and Background

Pyspark Environment

Data Preprocessing

Explanatory Data Analysis

Categorical Data

Numerical Variables

Data cleaning for text classification

Nature Language Processing on user descriptions and tweets: Tokens

Training, Testing, and Validation Dataset

Multinomial Regression Model

Random Forest

Neural Network with Keras API

Prediction

Conclusion and Possible Improvement

Tranditional Statistical Learning: Classification in Self-Assessed Financial Health Status

Disclaimer

Video

Background

Highlights

Exploratory Data Analysis

Feature Engineering

Random Foresting

Neural Network

Performance of the Model

Enviorment