如何在Ubuntu上安装内核对应的源码来编译驱动？

最近在学驱动模块，平台是ARM，因为课上老师讲的有点显浅，所以打算自己再深入研究一下。
所以在虚拟机里面装了ubuntu，打算用来深入学习。
目前的版本是

# uname -a
Linux ubuntu-ldm 4.4.0-28-generic #47-Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

我已经安装了以下的包

# dpkg -l | grep linux
ii  console-setup-linux                  1.108ubuntu15                            all          Linux specific part of console-setup
ii  libselinux1:amd64                    2.4-3build2                              amd64        SELinux runtime shared libraries
ii  linux-base                           4.0ubuntu1                               all          Linux image base package
ii  linux-firmware                       1.157.2                                  all          Firmware for Linux kernel drivers
ii  linux-generic                        4.4.0.28.30                              amd64        Complete Generic Linux kernel and headers
ii  linux-headers-4.4.0-28               4.4.0-28.47                              all          Header files related to Linux kernel version 4.4.0
ii  linux-headers-4.4.0-28-generic       4.4.0-28.47                              amd64        Linux kernel headers for version 4.4.0 on 64 bit x86 SMP
ii  linux-headers-generic                4.4.0.28.30                              amd64        Generic Linux kernel headers
ii  linux-image-4.4.0-28-generic         4.4.0-28.47                              amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-extra-4.4.0-28-generic   4.4.0-28.47                              amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-generic                  4.4.0.28.30                              amd64        Generic Linux kernel image
ii  linux-libc-dev:amd64                 4.4.0-28.47                              amd64        Linux Kernel Headers for development
ii  linux-sound-base                     1.0.25+dfsg-0ubuntu5                     all          base package for ALSA and OSS sound systems
ii  linux-source                         4.4.0.28.30                              all          Linux kernel source with Ubuntu patches
ii  linux-source-4.4.0                   4.4.0-28.47                              all          Linux kernel source for version 4.4.0 with Ubuntu patches
ii  util-linux                           2.27.1-6ubuntu3.1                        amd64        miscellaneous system utilities

并且将/usr/src/linux-source-4.4.0.tar.bz2解压到/home/test/WorkSpace/Kernel/linux-source-4.4.0下
然后写了一个小小的测试程序

#include <linux/init.h>
#include <linux/module.h>

MODULE_LICENSE("Dual BSD/GPL");

static int hello_init (void) {
        printk (KERN_ALERT "Hello, World\n");
        return 0;
}

static void hello_exit (void) {
        printk (KERN_ALERT "Goodbye, cruel world\n");
}

module_init (hello_init);
module_exit (hello_exit);

Makefile

obj-m += module.o

module-objs := hello.o

all:
        make modules M=`pwd` -C /home/test/WorkSpace/Kernel/linux-source-4.4.0

clean:
        make modules clean M=`pwd` -C /home/test/WorkSpace/Kernel/linux-source-4.4.0

令我不解的是

# sudo insmod module.ko
insmod: ERROR: could not insert module module.ko: Invalid module format

实在想不通为何没法装载这个内核模块？
是因为我用错了版本的源码嘛？
难道是要用自己编译的uImage来启动系统才可以嘛？
要如何自己手动编译替换Ubuntu的内核？