deferred work

背景信息

延期工作是内核的一个特色，用于在随后的某个时间执行某个代码。这个被预定的代码可以运行在进程上下文或者中断上下文。延期工作被用于完成中断处理功能因为中断拥有重要的要求和限制如下：

• 中断处理程序的执行时间要尽可能地小
• 在中断上下文中不能够使用导致阻塞的调用

使用延期工作

Softirqs

softirqs不能够被设备驱动使用，他们被保留用于多种多样的内核子系统。因为固定数目的softirqs被在编译时被定义。

• HI_SOFTIRQ and TASKLET_SOFTIRQ - running tasklets
• TIMER_SOFTIRQ - running timers
• NET_TX_SOFIRQ and NET_RX_SOFTIRQ - used by the networking subsystem
• BLOCK_SOFTIRQ - used by the IO subsystem
• BLOCK_IOPOLL_SOFTIRQ - used by the IO subsystem to increase performance when the iopoll handler is invoked;
• SCHED_SOFTIRQ - load balancing
• HRTIMER_SOFTIRQ - implementation of high precision timers
• RCU_SOFTIRQ - implementation of RCU type mechanisms

Softirqs运行在中断上下文当中，这意味着它们不能调用导致阻塞的方法。如果softirq处理器中调用了可能导致阻塞的方法，那么工作队列将会计划去执行这些导致阻塞的调用。

Tasklets

tasklet是运行在中断上下文中的延期工作的一种特殊的形式，就像是softirq一样。在softirqs和tasklets之间主要的不同是tasklets可以被动态地进行分配，所以它可以被应用到设备驱动程序当中。tasklet的代表是struct tasklet，并且在使用它之前需要首先进行初始化。在初始化之前的tasklet可以被定义为如下这样：

void handler(unsigned long data);

DECLARE_TASKLET(tasklet, handler, data);
DECLARE_TASKLET_DISABLED(tasklet, handler, data);

如果我们想要手动初始化tasklet，那么我们可以如下这样做：

void handler(unsigned long data);

struct tasklet_struct tasklet;

tasklet_init(&tasklet, handler, data);

当被执行的时候，dada会被送到handler当中。
讲tasklet运行其阿里需要调用schedule，tasklets运行自softirqs

void tasklet_schedule(struct tasklet_struct *tasklet);

void tasklet_hi_schedule(struct tasklet_struct *tasklet);

当使用tasklet_schedule，TASKLET_SOFTIRQ softirq 被计划安排并且所有的tasklet被计划运行。

Tasklets可以别掩盖，下面的方法可以使用

void tasklet_enable(struct tasklet_struct * tasklet );
void tasklet_disable(struct tasklet_struct * tasklet );

Timers

一个特殊类型的延期工作，经常被使用，就是timers。他们被struct timer_list定义。运行在中断上下文并softirq实现。在使用timer之前，调用timer_setup()来完成初始化的工作。

#include <linux / sched.h>

void timer_setup(struct timer_list * timer,
                 void (*function)(struct timer_list *),
                 unsigned int flags);

作为参数的function是一个handler

schedule a timeer is done with mod_timer

int mod_timer(struct timer_list *timer, unsigned long expires);

timers的时间单位是jiffie。jiffie依赖于具体的平台。为了在jiffie和second之间进行转换，可以按照如下的公式

jiffies_value = seconds_value * HZ ;
seconds_value = jiffies_value / HZ ;

初始化并且定时一秒的常用的代码顺序是

#include <linux/sched.h>

void timer_function(struct timer_list *);

struct timer_list timer ;
unsigned long seconds = 1;

timer_setup(&timer, timer_function, 0);
mod_timer(&timer, jiffies + seconds * HZ);

如何停止它呢?

del_timer_sync(&timer);

锁

为了使在进程上下文（A）中运行的代码与在softirq上下文（B）中运行的代码之间同步，我们需要使用特殊的锁定原语。 在（A）中，我们必须使用自旋锁操作来增强当前处理器下半部处理程序的停用，而在（B）中，仅使用基本自旋锁操作。 使用自旋锁可确保在多个CPU之间不会出现竞争，而停用softirq则可确保我们不会在已获得自旋锁的同一CPU上调度softirq的死锁。

我们可以使用local_bh_disable（）和local_bh_enable（）来禁用和启用softirqs处理程序（并且由于它们运行在softirqs之上，因此它们还包括计时器和tasklet）：

void local_bh_disable(void);
void local_bh_enable(void);

允许嵌套调用，仅当所有local_bh_disable（）调用均已由local_bh_enable（）调用进行补充时，才可以实际重新激活softirqs：

/* We assume that softirqs are enabled */
local_bh_disable();  /* Softirqs are now disabled */
local_bh_disable();  /* Softirqs remain disabled */

local_bh_enable();  /* Softirqs remain disabled */
local_bh_enable();  /* Softirqs are now enabled */

Attention
These above calls will disable the softirqs only on the local processor and they are usually not safe to use, they must be complemented with spinlocks.

Most of the time device drivers will use special versions of spinlocks calls for synchronization like spin_lock_bh() and spin_unlock_bh():

void spin_lock_bh(spinlock_t *lock);
void spin_unlock_bh(spinlock_t *lock);

工作队列

#include <linux/workqueue.h>

DECLARE_WORK(name , void (*function)(struct work_struct *));
DECLARE_DELAYED_WORK(name, void(*function)(struct work_struct *));

INIT_WORK(struct work_struct *work, void(*function)(struct work_struct *));
INIT_DELAYED_WORK(struct delayed_work *work, void(*function)(struct work_struct *));

DECLARE_WORK() and DECLARE_DELAYED_WORK() declare and initialize a work item, and INIT_WORK() and INIT_DELAYED_WORK() initialize an already declared work item.

定义并且初始化

#include <linux/workqueue.h>

void my_work_handler(struct work_struct *work);

DECLARE_WORK(my_work, my_work_handler);

初始化之前已经定义过了的

void my_work_handler(struct work_struct * work);

struct work_struct my_work;

INIT_WORK(&my_work, my_work_handler);

在前面已经进行了定义和初始化，那么我们如何来使用呢？

schedule_work(struct work_struct *work);

schedule_delayed_work(struct delayed_work *work, unsigned long delay);

我们也可以对工作进行撤销

int cancel_work_sync(struct delayed_work *work);
int cancel_delayed_work_sync(struct delayed_work *work);

We can wait for a workqueue to complete running all of its work items by calling flush_scheduled_work():

void flush_scheduled_work(void);

Finally, the following functions can be used to schedule work items on a particular processor (schedule_delayed_work_on()), or on all processors (schedule_on_each_cpu()):

int schedule_delayed_work_on(int cpu, struct delayed_work *work, unsigned long delay);
int schedule_on_each_cpu(void(*function)(struct work_struct *));

A usual sequence to initialize and schedule a work item is the following:

void my_work_handler(struct work_struct *work);

struct work_struct my_work;

INIT_WORK(&my_work, my_work_handler);

schedule_work(&my_work);

my_work_handler方法接收task作为参数。为了访问模块的私有数据，可以使用container_of

struct my_device_data {
    struct work_struct my_work;
    // ...
};

void my_work_handler(struct work_struct *work)
{
   structure my_device_data * my_data;

   my_data = container_of(work, struct my_device_data,  my_work);
   // ...
}

拥有上面方法的计划工作会在被称为events/x的内核线程上下文中运行handler。x是处理器号，内核会为每个处理器初始化一个内核线程。

create_workqueue() uses one thread for each processor in the system, and create_singlethread_workqueue() uses a single thread.
在队列中添加工作，使用queue_work或者queue_delayed_work方法

int queue_work(struct workqueue_struct * queue, struct work_struct *work);

int queue_delayed_work(struct workqueue_struct *queue,
                       struct delayed_work * work , unsigned long delay);

接着定义和初始化另外的工作队列，定义和初始化工作条目，并且将它添加到队列中

void my_work_handler(struct work_struct *work);

struct work_struct my_work;
struct workqueue_struct * my_workqueue;

my_workqueue = create_singlethread_workqueue("my_workqueue");
INIT_WORK(&my_work, my_work_handler);

queue_work(my_workqueue, &my_work);

接下来演示如何移除工作队列

flush_workqueue(my_workqueue);
destroy_workqueue(my_workqueue);

内核线程

内核线程的出现正是为了满足在进程的上下文中运行内核的代码。内核线程是工作队列机制的基石。从本质上来讲，内核线程是仅仅运行在内核模式的线程并且没有用户地址空间或者其他的属性。使用kthread_create()创建内核线程

#include <linux/kthread.h>

structure task_struct *kthread_create(int (*threadfn)(void *data),
                                      void *data, const char namefmt[], ...);

threadfn is a function that will be run by the kernel thread
data is a parameter to be sent to the function
namefmt represents the kernel thread name, as it is displayed in ps/top ; Can contain sequences %d , %s etc. Which will be replaced according to the standard printf syntax.

kthread_create (f, NULL, "%skthread%d", "my", 0);

The kernel thread created with this function will be stopped (in the TASK_INTERRUPTIBLE state). To start the kernel thread, call the wake_up_process():

#include <linux/sched.h>

int wake_up_process(struct task_struct *p);

使用kthread_run()来创建并运行内核线程

struct task_struct * kthread_run(int (*threadfn)(void *data)
                                 void *data, const char namefmt[], ...);

• can’t access the user address space (even with copy_from_user, copy_to_user) because a thread kernel does not have a user address space
• can’t implement busy wait code that runs for a long time; if the kernel is compiled without the preemptive option, that code will run without being preempted by other kernel threads or user processes thus hogging the system
• can call blocking operations
• can use spinlocks, but if the hold time of the lock is significant, it is recommended to use mutexes

The termination of a thread kernel is done voluntarily, within the function running in the thread kernel, by calling do_exit():

fastcall NORET_TYPE void do_exit(long code);

大多数内核线程的实现使用相同的模型，我们也建议这样做，这样可以避免共同的错误

#include <linux/kthread.h>

DECLARE_WAIT_QUEUE_HEAD(wq);

// list events to be processed by kernel thread
structure list_head events_list;
struct spin_lock events_lock;


// structure describing the event to be processed
struct event {
    struct list_head lh;
    bool stop;
    //...
};

struct event* get_next_event(void)
{
    struct event *e;

    spin_lock(&events_lock);
    e = list_first_entry(&events_list, struct event*, lh);
    if (e)
        list_del(&events->lh);
    spin_unlock(&events_lock);

    return e
}

int my_thread_f(void *data)
{
    struct event *e;

    while (true) {
        wait_event(wq, (e = get_next_event));

        /* Event processing */

        if (e->stop)
            break;
    }

    do_exit(0);
}

/* start and start kthread */
kthread_run(my_thread_f, NULL, "%skthread%d", "my", 0);

void send_event(struct event *ev)
{
    spin_lock(&events_lock);
    list_add(&ev->lh, &list_events);
    spin_unlock(events_lock);
    wake_up(&wq);
}

/*
 * Deferred Work
 *
 * Exercise #1, #2: simple timer
 */

#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/module.h>
#include <linux/sched.h>

MODULE_DESCRIPTION("Simple kernel timer");
MODULE_AUTHOR("SO2");
MODULE_LICENSE("GPL");

#define TIMER_TIMEOUT	1

static struct timer_list timer;

static void timer_handler(struct timer_list *tl)
{
	/* TODO 1/4: print a message */
	static size_t nseconds;

	nseconds += TIMER_TIMEOUT;
	pr_info("[timer_handler] nseconds = %d\n", nseconds);

	/* TODO 2: rechedule timer */
	mod_timer(tl, jiffies + TIMER_TIMEOUT * HZ);
}

static int __init timer_init(void)
{
	pr_info("[timer_init] Init module\n");

	/* TODO 1: initialize timer */
	timer_setup(&timer, timer_handler, 0);

	/* TODO 1: schedule timer for the first time */
	mod_timer(&timer, jiffies + TIMER_TIMEOUT * HZ);

	return 0;
}

static void __exit timer_exit(void)
{
	pr_info("[timer_exit] Exit module\n");

	/* TODO 1: cleanup; make sure the timer is not running after we exit */
	del_timer_sync(&timer);
}

module_init(timer_init);
module_exit(timer_exit);

/*
 * SO2 - Lab 6 - Deferred Work
 *
 * Exercises #3, #4, #5: deferred work
 *
 * Code skeleton.
 */

#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/module.h>
#include <linux/fs.h>
#include <linux/cdev.h>
#include <linux/sched.h>
#include <linux/slab.h>
#include <linux/uaccess.h>
#include <linux/sched/task.h>
#include "../include/deferred.h"

#define MY_MAJOR		42
#define MY_MINOR		0
#define MODULE_NAME		"deferred"

#define TIMER_TYPE_NONE		-1
#define TIMER_TYPE_SET		0
#define TIMER_TYPE_ALLOC	1
#define TIMER_TYPE_MON		2

MODULE_DESCRIPTION("Deferred work character device");
MODULE_AUTHOR("SO2");
MODULE_LICENSE("GPL");

struct mon_proc {
	struct task_struct *task;
	struct list_head list;
};

static struct my_device_data {
	struct cdev cdev;
	/* TODO 1: add timer */
	struct timer_list timer;
	/* TODO 2: add flag */
	int flag;
	/* TODO 3: add work */
	struct work_struct work;
	/* TODO 4: add list for monitored processes */
	struct list_head list;
	/* TODO 4: add spinlock to protect list */
	spinlock_t lock;
} dev;

static void alloc_io(void)
{
	set_current_state(TASK_INTERRUPTIBLE);
	schedule_timeout(5 * HZ);
	pr_info("Yawn! I've been sleeping for 5 seconds.\n");
}

static struct mon_proc *get_proc(pid_t pid)
{
	struct task_struct *task;
	struct mon_proc *p;

	task = pid_task(find_vpid(pid), PIDTYPE_PID);
	if (!task)
		return ERR_PTR(-ESRCH);

	p = kmalloc(sizeof(p), GFP_ATOMIC);
	if (!p)
		return ERR_PTR(-ENOMEM);

	get_task_struct(task);
	p->task = task;

	return p;
}


/* TODO 3/4: define work handler */
static void work_handler(struct work_struct *work)
{
	alloc_io();
}

#define ALLOC_IO_DIRECT
/* TODO 3: undef ALLOC_IO_DIRECT*/
#undef ALLOC_IO_DIRECT

static void timer_handler(struct timer_list *tl)
{
	/* TODO 1/44: implement timer handler */
	struct my_device_data *my_data = from_timer(my_data, tl, timer);

	pr_info("[timer_handler] pid = %d, comm = %s\n",
		current->pid, current->comm);

	/* TODO 2/38: check flags: TIMER_TYPE_SET or TIMER_TYPE_ALLOC */
	switch (my_data->flag) {
	case TIMER_TYPE_SET:
		break;
	case TIMER_TYPE_ALLOC:
#ifdef ALLOC_IO_DIRECT
		alloc_io();
#else
		/* TODO 3: schedule work */
		schedule_work(&my_data->work);
#endif
		break;
	case TIMER_TYPE_MON:
	{
		/* TODO 4/19: iterate the list and check the proccess state */
		struct mon_proc *p, *n;

		spin_lock(&my_data->lock);
		list_for_each_entry_safe(p, n, &my_data->list, list) {
			/* TODO 4: if task is dead print info ... */
			/* TODO 4: ... decrement task usage counter ... */
			/* TODO 4: ... remove it from the list ... */
			/* TODO 4: ... free the struct mon_proc */
			if (p->task->state == TASK_DEAD) {
				pr_info("task %s (%d) is dead\n", p->task->comm,
					p->task->pid);
				put_task_struct(p->task);
				list_del(&p->list);
				kfree(p);
			}
		}
		spin_unlock(&my_data->lock);

		mod_timer(&my_data->timer, jiffies + HZ);
		break;
	}
	default:
		break;
	}
}

static int deferred_open(struct inode *inode, struct file *file)
{
	struct my_device_data *my_data =
		container_of(inode->i_cdev, struct my_device_data, cdev);
	file->private_data = my_data;
	pr_info("[deferred_open] Device opened\n");
	return 0;
}

static int deferred_release(struct inode *inode, struct file *file)
{
	pr_info("[deferred_release] Device released\n");
	return 0;
}

static long deferred_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
{
	struct my_device_data *my_data = (struct my_device_data*) file->private_data;

	pr_info("[deferred_ioctl] Command: %s\n", ioctl_command_to_string(cmd));

	switch (cmd) {
		case MY_IOCTL_TIMER_SET:
			/* TODO 2: set flag */
			my_data->flag = TIMER_TYPE_SET;
			/* TODO 1: schedule timer */
			mod_timer(&my_data->timer, jiffies + arg * HZ);
			break;
		case MY_IOCTL_TIMER_CANCEL:
			/* TODO 1: cancel timer */
			del_timer(&my_data->timer);
			break;
		case MY_IOCTL_TIMER_ALLOC:
			/* TODO 2/2: set flag and schedule timer */
			my_data->flag = TIMER_TYPE_ALLOC;
			mod_timer(&my_data->timer, jiffies + arg * HZ);
			break;
		case MY_IOCTL_TIMER_MON:
		{
			/* TODO 4/8: use get_proc() and add task to list */
			struct mon_proc *p = get_proc(arg);
			if (IS_ERR(p))
				return PTR_ERR(p);

			/* TODO 4: protect access to list */
			spin_lock_bh(&my_data->lock);
			list_add(&p->list, &my_data->list);
			spin_unlock_bh(&my_data->lock);

			/* TODO 4/2: set flag and schedule timer */
			my_data->flag = TIMER_TYPE_MON;
			mod_timer(&my_data->timer, jiffies + HZ);
			break;
		}
		default:
			return -ENOTTY;
	}
	return 0;
}

struct file_operations my_fops = {
	.owner = THIS_MODULE,
	.open = deferred_open,
	.release = deferred_release,
	.unlocked_ioctl = deferred_ioctl,
};

static int deferred_init(void)
{
	int err;

	pr_info("[deferred_init] Init module\n");
	err = register_chrdev_region(MKDEV(MY_MAJOR, MY_MINOR), 1, MODULE_NAME);
	if (err) {
		pr_info("[deffered_init] register_chrdev_region: %d\n", err);
		return err;
	}

	/* TODO 2: Initialize flag. */
	dev.flag = TIMER_TYPE_NONE;
	/* TODO 3: Initialize work. */
	INIT_WORK(&dev.work, work_handler);

	/* TODO 4/2: Initialize lock and list. */
	spin_lock_init(&dev.lock);
	INIT_LIST_HEAD(&dev.list);

	cdev_init(&dev.cdev, &my_fops);
	cdev_add(&dev.cdev, MKDEV(MY_MAJOR, MY_MINOR), 1);

	/* TODO 1: Initialize timer. */
	timer_setup(&dev.timer, timer_handler, 0);

	return 0;
}

static void deferred_exit(void)
{
	struct mon_proc *p, *n;

	pr_info("[deferred_exit] Exit module\n" );

	cdev_del(&dev.cdev);
	unregister_chrdev_region(MKDEV(MY_MAJOR, MY_MINOR), 1);

	/* TODO 1: Cleanup: make sure the timer is not running after exiting. */
	del_timer_sync(&dev.timer);
	/* TODO 3: Cleanup: make sure the work handler is not scheduled. */
	flush_scheduled_work();

	/* TODO 4/8: Cleanup the monitered process list */
	list_for_each_entry_safe(p, n, &dev.list, list) {
		/* TODO 4: ... decrement task usage counter ... */
		/* TODO 4: ... remove it from the list ... */
		/* TODO 4: ... free the struct mon_proc */
		put_task_struct(p->task);
		list_del(&p->list);
		kfree(p);
	}
}

module_init(deferred_init);
module_exit(deferred_exit);

/*
 * SO2 - Lab 6 - Deferred Work
 *
 * Exercise #6: kernel thread
 */

#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/module.h>
#include <linux/sched.h>
#include <asm/atomic.h>
#include <linux/kthread.h>

MODULE_DESCRIPTION("Simple kernel thread");
MODULE_AUTHOR("SO2");
MODULE_LICENSE("GPL");

wait_queue_head_t wq_stop_thread;
atomic_t flag_stop_thread;
wait_queue_head_t wq_thread_terminated;
atomic_t flag_thread_terminated;


int my_thread_f(void *data)
{
	pr_info("[my_thread_f] Current process id is %d (%s)\n",
		current->pid, current->comm);
	/* TODO: Wait for command to remove module on wq_stop_thread queue. */
	wait_event_interruptible(wq_stop_thread, atomic_read(&flag_stop_thread) != 0);

	/* TODO: set flag to mark kernel thread termination */
	atomic_set(&flag_thread_terminated, 1);
	/* TODO: notify the unload process that we have exited */
	wake_up_interruptible(&wq_thread_terminated);
	pr_info("[my_thread_f] Exiting\n");
	do_exit(0);
}

static int __init kthread_init(void)
{
	pr_info("[kthread_init] Init module\n");

	/* TODO/4: init the waitqueues and flags */
	init_waitqueue_head(&wq_stop_thread);
	atomic_set(&flag_stop_thread, 0);
	init_waitqueue_head(&wq_thread_terminated);
	atomic_set(&flag_thread_terminated, 0);
	/* TODO: create and start the kernel thread */
	kthread_run(my_thread_f, NULL, "%skthread%d", "my", 0);

	return 0;
}

static void __exit kthread_exit(void)
{
	/* TODO/2: notify the kernel thread that its time to exit */
	atomic_set(&flag_stop_thread, 1);
	wake_up_interruptible(&wq_stop_thread);
	/* TODO: wait for the kernel thread to exit */
	wait_event_interruptible(wq_thread_terminated, atomic_read(&flag_thread_terminated) != 0);
	pr_info("[kthread_exit] Exit module\n");
}

module_init(kthread_init);
module_exit(kthread_exit);

来源：CSDN

作者：久许

链接：https://blog.csdn.net/jiuweideqixu/article/details/104138841