{"id":473,"date":"2016-04-24T23:58:25","date_gmt":"2016-04-24T21:58:25","guid":{"rendered":"http:\/\/tomaszkacmajor.pl\/?p=473"},"modified":"2020-05-17T16:30:56","modified_gmt":"2020-05-17T14:30:56","slug":"svm-model-selection","status":"publish","type":"post","link":"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/","title":{"rendered":"SVM model selection – how to adjust all these knobs pt. 1"},"content":{"rendered":"\n

This time we are going to discuss the influence of two basic variables on the quality of SVM classifier. They are called hyperparameters<\/em> to distinguish them from the parameters<\/em> optimized in a machine learning procedures. Two previous posts introduced Support Vector Machine<\/a> itself and data preprocessing<\/a> for this classifier. As in other Machine Learning techniques there is also a need to properly adjust some system variables to find the best model for our needs. Here, we will focus on description of complexity parameter and gamma parameter from the Gaussian kernel. In the next article we will find an optimum SVM model for the foreground\/background estimation problem in Flover<\/a> project using model validation techniques.
<\/p>\n\n\n\n\n\n\n\n

Complexity parameter<\/h2>\n\n\n\n

Let’s introduce the first hyperparameter of an SVM classifier. It defines its complexity and is also known as regularization or softness parameter indicated by \"C\". This value tells us how many points from the training set are allowed to be on the wrong side of the margin. SVM is always looking for the largest available margin between data groups. The margin is defined as a distance between decision hyperplane and the nearest point from a group, ie. the Support Vector. If all training data points are properly classified, the margin is usually relatively thin so it may be called a hard-margin<\/strong>. But we often deal with a set which is not linearly separable or have some outliers. Then, it’s better to soften<\/strong> the margin and allow for some misclassification in exchange to the grouping generalization. Below is the 2D example of two classes of points which are not linearly separable.<\/p>\n\n\n\n

\"soft_margin<\/a><\/figure><\/div>\n\n\n\n

We see two red points which are inside the margin or even wrongly classified. How do we tell a classifier that we accept such soft-margin?
Decision hyperplane is defined by weigth vector \"\mathbf{w}\" and bias \"\mathbf{b}\". It’s composed from such \"\mathbf{x}\" points where \"f(\mathbf{x})=\mathbf{w}^T\mathbf{x}+\mathbf{b}=0\". So, it’s simply dividing the plane into two regions. On one side \"f(\mathbf{x})>0\", on another side \"f(\mathbf{x})<0\". Now, to find the widest possible margin which have direct impact on a decision hyperplane the following cost function has to be minimized. Parameters \"\mathbf{w}\" and \"\mathbf{b}\" are the searched variables but \"\mathbf{b}\" comes in some further constraints which are not discussed here.<\/p>\n\n\n\n

\"{\frac{1}{2} \mathbf{ w}^T \mathbf{ w}+ {C} \sum_{i=1}^{l} \xi_i}\"<\/p>\n\n\n\n

The second term is a penalty of violating data<\/strong>. It takes all points which are on the wrong side of the margin and sums slack variables<\/em> \"\xi_i\" assigned to them (\"0\leq\xi_i\leq1\" – points located in the margin, \"\xi_i>1\" – misclassified points). If the complexity parameter \"C\" is high, we get a hard margin. Otherwise, if \"C\" is close to \"0\", the SVM margin is very soft and almost not caring about proper division of data. Therefore, the decision about the adequate complexity parameter is very important to obtain a good SVM model.<\/p>\n\n\n\n

Gamma parameter<\/h2>\n\n\n\n

In many real-life cases a linear separation of our dataset is not efficient enough. Like presented in one of previous posts about SVM<\/a>, one can introduce a kernel<\/strong> functions which transforms the data into higher dimensions. Each of these kernels depend on their own parameters. It may be, for instance, a degree of a polynomial kernel or gamma\/sigma variable of one of the most popular kernel – a gaussian function. Let’s recall how the \"\gamma\" parameter is influencing the width of a Gaussian.<\/p>\n\n\n\n

\"gaussian_curves\"<\/figure><\/div>\n\n\n\n
<\/div>\n\n\n\n

It’s clear that the larger gamma the thinner gaussian curve is. Gaussian kernel in SVM is described as below<\/p>\n\n\n\n

\"{k(\textbf{x},\textbf{x'})= e^{- \gamma \| \textbf{x} - \textbf{x'} \| ^2} }\"<\/p>\n\n\n\n

Details of the kernel trick<\/em> and other foundations of SVM theory are very well described in [7]<\/a>. In short, gaussian kerneling for a point \"\mathbf{x}\" consists in calculation of a sum of Gaussian “bumps” centered around each point \"\mathbf{x'}\" in a training dataset. When \"\gamma\" is small a given data point \"\mathbf{x}\" has a non-zero kernel value relative to any point in the set. Therefore, the whole set of support vectors affects the value of the discriminant function at \"\mathbf{x}\", resulting in a smooth decision boundary. This situation is depicted on top-left panel below. As \"\gamma\" is increased the locality of the support vector expansion increases, leading to greater curvature<\/strong> of the decision boundary. It can be observed on the last panel in the below example.<\/p>\n\n\n\n

\"gaussian_gamma\"<\/a><\/figure><\/div>\n\n\n\n

The most dangerous and common effect of increasing \"\gamma\" is overfitting<\/strong>. It appears when the SVM model is extremely well trained for the training data. But it doesn’t generalize well, i.e. it’s not prepared to respond well for the new, unknown set of data.<\/p>\n\n\n\n

Summary<\/h2>\n\n\n\n

Two hyperparameters which decide about SVM model were presented. Complexity \"C\" controls the softness of the decision margin. Gaussian kernel bandwidth \"\gamma\" controls the curvature of a hyperplane. In the next post I will describe how to find them based on a specific SVM case.<\/p>\n\n\n\n

I strongly encourage you to play with these parameters here<\/a>. You can place some points in 2D space and group them using an SVM model with different parameters (-c for complexity and -g for gamma).<\/p><\/blockquote>\n\n\n\n

Sources:
1. Presentation “Practical guide to SVM<\/a>” by Tingfan Wu.
2. SVM toy<\/a> based on libsvm library for Python.
3. SVM tutorial<\/a> by Jason Weston.
4. Documentation <\/a>of Shark Machine Learning library for C++.
5. Documentation <\/a>of scikit-learn library for Python.
6. Topic about SVM variables on StackExchange<\/a>.
7. Ben-Hur, Weston – “A User\u2019s Guide to Support Vector Machines<\/a>“.<\/p>\n","protected":false},"excerpt":{"rendered":"

This time we are going to discuss the influence of two basic variables on the quality of SVM classifier. They are called hyperparameters to distinguish them from the parameters optimized in a machine learning procedures. Two previous posts introduced Support Vector Machine itself and data preprocessing for this classifier. As in other Machine Learning techniques […]<\/p>\n","protected":false},"author":1,"featured_media":526,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[17,12,11,14,15],"class_list":["post-473","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-flover-project","tag-cross-validation","tag-daj-sie-poznac","tag-flover","tag-machine-learning","tag-svm"],"blocksy_meta":[],"yoast_head":"\nSVM model selection - how to adjust all these knobs pt. 1 - tomaszkacmajor.pl<\/title>\n<meta name=\"description\" content=\"Basic SVM parameters like complexity, gaussian kernel gamma are discussed in order to find the best SVM model for a given data.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"SVM model selection - how to adjust all these knobs pt. 1 - tomaszkacmajor.pl\" \/>\n<meta property=\"og:description\" content=\"Basic SVM parameters like complexity, gaussian kernel gamma are discussed in order to find the best SVM model for a given data.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/\" \/>\n<meta property=\"og:site_name\" content=\"tomaszkacmajor.pl\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/ProggBlogg\/\" \/>\n<meta property=\"article:published_time\" content=\"2016-04-24T21:58:25+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-05-17T14:30:56+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/tomaszkacmajor.pl\/wp-content\/uploads\/2016\/04\/SVM-Model-Selection-Methods.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"2560\" \/>\n\t<meta property=\"og:image:height\" content=\"1024\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"tomasz.kacmajor\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@tkacmajor\" \/>\n<meta name=\"twitter:site\" content=\"@tkacmajor\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"tomasz.kacmajor\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/\",\"url\":\"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/\",\"name\":\"SVM model selection - how to adjust all these knobs pt. 1 - tomaszkacmajor.pl\",\"isPartOf\":{\"@id\":\"https:\/\/tomaszkacmajor.pl\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/tomaszkacmajor.pl\/wp-content\/uploads\/2016\/04\/SVM-Model-Selection-Methods.jpg\",\"datePublished\":\"2016-04-24T21:58:25+00:00\",\"dateModified\":\"2020-05-17T14:30:56+00:00\",\"author\":{\"@id\":\"https:\/\/tomaszkacmajor.pl\/#\/schema\/person\/5f40890309a32ae4f63fa6a284215b6c\"},\"description\":\"Basic SVM parameters like complexity, gaussian kernel gamma are discussed in order to find the best SVM model for a given data.\",\"breadcrumb\":{\"@id\":\"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/#primaryimage\",\"url\":\"https:\/\/tomaszkacmajor.pl\/wp-content\/uploads\/2016\/04\/SVM-Model-Selection-Methods.jpg\",\"contentUrl\":\"https:\/\/tomaszkacmajor.pl\/wp-content\/uploads\/2016\/04\/SVM-Model-Selection-Methods.jpg\",\"width\":2560,\"height\":1024},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Strona g\u0142\u00f3wna\",\"item\":\"https:\/\/tomaszkacmajor.pl\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"SVM model selection – how to adjust all these knobs pt. 1\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/tomaszkacmajor.pl\/#website\",\"url\":\"https:\/\/tomaszkacmajor.pl\/\",\"name\":\"tomaszkacmajor.pl\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/tomaszkacmajor.pl\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/tomaszkacmajor.pl\/#\/schema\/person\/5f40890309a32ae4f63fa6a284215b6c\",\"name\":\"tomasz.kacmajor\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/tomaszkacmajor.pl\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a1fe5c8a80549b9a680c7a6f9ea33a94?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a1fe5c8a80549b9a680c7a6f9ea33a94?s=96&d=mm&r=g\",\"caption\":\"tomasz.kacmajor\"},\"url\":\"https:\/\/tomaszkacmajor.pl\/index.php\/author\/tomasz-kacmajor\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"SVM model selection - how to adjust all these knobs pt. 1 - tomaszkacmajor.pl","description":"Basic SVM parameters like complexity, gaussian kernel gamma are discussed in order to find the best SVM model for a given data.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/","og_locale":"en_US","og_type":"article","og_title":"SVM model selection - how to adjust all these knobs pt. 1 - tomaszkacmajor.pl","og_description":"Basic SVM parameters like complexity, gaussian kernel gamma are discussed in order to find the best SVM model for a given data.","og_url":"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/","og_site_name":"tomaszkacmajor.pl","article_publisher":"https:\/\/www.facebook.com\/ProggBlogg\/","article_published_time":"2016-04-24T21:58:25+00:00","article_modified_time":"2020-05-17T14:30:56+00:00","og_image":[{"width":2560,"height":1024,"url":"https:\/\/tomaszkacmajor.pl\/wp-content\/uploads\/2016\/04\/SVM-Model-Selection-Methods.jpg","type":"image\/jpeg"}],"author":"tomasz.kacmajor","twitter_card":"summary_large_image","twitter_creator":"@tkacmajor","twitter_site":"@tkacmajor","twitter_misc":{"Written by":"tomasz.kacmajor","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/","url":"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/","name":"SVM model selection - how to adjust all these knobs pt. 1 - tomaszkacmajor.pl","isPartOf":{"@id":"https:\/\/tomaszkacmajor.pl\/#website"},"primaryImageOfPage":{"@id":"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/#primaryimage"},"image":{"@id":"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/#primaryimage"},"thumbnailUrl":"https:\/\/tomaszkacmajor.pl\/wp-content\/uploads\/2016\/04\/SVM-Model-Selection-Methods.jpg","datePublished":"2016-04-24T21:58:25+00:00","dateModified":"2020-05-17T14:30:56+00:00","author":{"@id":"https:\/\/tomaszkacmajor.pl\/#\/schema\/person\/5f40890309a32ae4f63fa6a284215b6c"},"description":"Basic SVM parameters like complexity, gaussian kernel gamma are discussed in order to find the best SVM model for a given data.","breadcrumb":{"@id":"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/#primaryimage","url":"https:\/\/tomaszkacmajor.pl\/wp-content\/uploads\/2016\/04\/SVM-Model-Selection-Methods.jpg","contentUrl":"https:\/\/tomaszkacmajor.pl\/wp-content\/uploads\/2016\/04\/SVM-Model-Selection-Methods.jpg","width":2560,"height":1024},{"@type":"BreadcrumbList","@id":"https:\/\/tomaszkacmajor.pl\/index.php\/2016\/04\/24\/svm-model-selection\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Strona g\u0142\u00f3wna","item":"https:\/\/tomaszkacmajor.pl\/"},{"@type":"ListItem","position":2,"name":"SVM model selection – how to adjust all these knobs pt. 1"}]},{"@type":"WebSite","@id":"https:\/\/tomaszkacmajor.pl\/#website","url":"https:\/\/tomaszkacmajor.pl\/","name":"tomaszkacmajor.pl","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/tomaszkacmajor.pl\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/tomaszkacmajor.pl\/#\/schema\/person\/5f40890309a32ae4f63fa6a284215b6c","name":"tomasz.kacmajor","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/tomaszkacmajor.pl\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/a1fe5c8a80549b9a680c7a6f9ea33a94?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a1fe5c8a80549b9a680c7a6f9ea33a94?s=96&d=mm&r=g","caption":"tomasz.kacmajor"},"url":"https:\/\/tomaszkacmajor.pl\/index.php\/author\/tomasz-kacmajor\/"}]}},"_links":{"self":[{"href":"https:\/\/tomaszkacmajor.pl\/index.php\/wp-json\/wp\/v2\/posts\/473","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tomaszkacmajor.pl\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tomaszkacmajor.pl\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tomaszkacmajor.pl\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tomaszkacmajor.pl\/index.php\/wp-json\/wp\/v2\/comments?post=473"}],"version-history":[{"count":21,"href":"https:\/\/tomaszkacmajor.pl\/index.php\/wp-json\/wp\/v2\/posts\/473\/revisions"}],"predecessor-version":[{"id":1711,"href":"https:\/\/tomaszkacmajor.pl\/index.php\/wp-json\/wp\/v2\/posts\/473\/revisions\/1711"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tomaszkacmajor.pl\/index.php\/wp-json\/wp\/v2\/media\/526"}],"wp:attachment":[{"href":"https:\/\/tomaszkacmajor.pl\/index.php\/wp-json\/wp\/v2\/media?parent=473"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tomaszkacmajor.pl\/index.php\/wp-json\/wp\/v2\/categories?post=473"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tomaszkacmajor.pl\/index.php\/wp-json\/wp\/v2\/tags?post=473"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}